Multivariate Methods for Heterogeneous Data — MSMB Chapter 9 (summary)

Own-words summary of an external source (see frontmatter for license/attribution). Not a substitute for the original chapter.

1. Scope

This chapter extends linear dimensionality reduction to heterogeneous and non-coordinate data: settings where observations are described not by a clean matrix of continuous variables but by pairwise distances or (dis)similarities, by categorical/contingency-table counts, by a mix of continuous and categorical variables plus contextual covariates, or by several data tables measured on the same samples. It equips a reader to build low-dimensional “maps” (ordinations) from distance matrices, to expose and visualize dependencies in categorical data, to detect hidden one-dimensional structure (gradients, orderings, “pseudotime”), to recognize and remove batch/nuisance effects, and to measure and exploit association between whole tables from different data domains. The treatment is deliberately applied and example-driven; it states methods and their behavior but often omits derivations.

2. Concepts & Methods

Multidimensional scaling (MDS) / Principal Coordinates Analysis (PCoA). Input: an $n\times n$ matrix of pairwise distances (or squared distances), not a feature matrix. Output: coordinates of points in a low-dimensional Euclidean space whose interpoint distances approximate the given ones, plus eigenvalues for a screeplot. Mechanism: form the squared-distance matrix (the chapter writes the elementwise product $D\bullet D$ , distinct from matrix products $DD$ or $D^2$ ); double-center it with the centering matrix $H = I - \tfrac{1}{n}\mathbf{1}\mathbf{1}^t$ and scale by $-\tfrac12$ to recover a cross-product (Gram) matrix $B = -\tfrac12\, H (D\bullet D)\, H$ ; eigendecompose $B = U\Lambda U^t$ ; take coordinates $\tilde X = U\Lambda^{1/2}$ , keeping the leading $r$ axes. This is the distance-space analogue of PCA’s best rank- $r$ approximation via SVD. Relation to PCA: when distances are Euclidean from an actual feature matrix, MDS reproduces PCA’s configuration; but MDS has only one mode (the sample points) — there is no dual variable space, hence no loadings and no biplot, which limits interpretation.
Non-metric MDS (NMDS). A rank-based, robust variant. Instead of matching distance values, it seeks a monotone transformation $f$ of the input dissimilarities $d$ and a low-dimensional configuration with distances $\tilde d$ so that $f(d)\approx\tilde d$ — i.e., only the ordering of distances must be preserved. Fit quality is the standardized residual sum of squares, the stress: $\text{stress}^2 = \frac{\sum (f(d)-\tilde d)^2}{\sum d^2}$ . Unlike MDS/PCA it is non-sequential: the target dimension $k$ must be fixed in advance, the optimization is iterative and randomly initialized, and there is no “percent variance per axis.”
Robustness (general principle). A method is robust if a few outliers cannot dominate it. Quantified by the breakdown point — the fraction of data that must be corrupted to move an estimate arbitrarily: $1/2$ for the median, only $1/n$ for the mean. Rank-based and nonparametric reductions inherit this robustness; NMDS is the ordination instance.
Supplementary information / supplementary points. Categorical covariates (batch, day, protocol, treatment, location, time) are folded into a continuous ordination by projecting the group mean (barycenter) of points sharing a level as an extra marker, without including those points in the matrix decomposition. This lets a few categorical variables annotate a PCA/MDS map and exposes nuisance structure such as batch effects.
Correspondence Analysis (CA) / dual scaling. Input: a two-way contingency table $\mathbf N$ of counts ( $I$ rows, $J$ columns, grand total $n_{\cdot\cdot}$ ). It decomposes the departure from independence the way PCA decomposes variance. Under independence each cell would be $n_{ij}=\frac{n_{i\cdot}}{n}\frac{n_{\cdot j}}{n}\,n$ (outer product of row and column marginal proportions); the total departure is the $\chi^2$ statistic $\mathcal X^2=\sum_{i,j}\frac{(n_{ij}-\text{expected})^2}{\text{expected}}$ , and CA is a weighted SVD/MDS using the $\chi^2$ distance in place of variance. Output: a symmetric biplot where rows and columns share status and are co-plotted; it has a barycentric property — each row point sits at the weighted center of gravity of the column points (weights = its conditional counts), so spatial proximity of a row level and a column level encodes strong co-occurrence. CA is the natural follow-up to a significant $\chi^2$ test, turning “not independent” into “which levels associate, positively or negatively.”
Ordination as latent-variable discovery. Just as clustering recovers a hidden categorical factor, ordination recovers a hidden continuous ordering or gradient. A horseshoe / arch in the first two coordinates is a signature of an underlying sequential latent variable (a gradient or chronology). Historically used for ecological gradients and seriation (ordering objects by presence/absence patterns, e.g. dating artifacts); modern analogue is recovering developmental trajectories / pseudotime in single-cell data.
Local / nonlinear embedding methods. When only local distances are trustworthy, downweight large distances. A radial-basis kernel $1-\exp\!\big(-\frac{d(x,y)^2}{\sigma^2}\big)$ saturates toward 1 for large $d$ , discounting far-apart pairs. t-SNE generalizes this by letting $\sigma^2$ vary locally (with a normalization), converting high-dimensional proximities into a probability distribution over point pairs and reconstructing a low-dimensional layout that reproduces those probabilities. Related families named: isomap, local linear embedding (LLE), Laplacian eigenmaps, diffusion maps (a Markov/affinity-kernel approach using local probabilistic transitions, then eigendecomposition), local principal curves, and principal curves (nonlinear analogue of a principal axis). Software named: Rtsne, RDRToolbox, diffusionMap, kernlab, LPCM, princurve.
Multitable / co-variation measures. Generalize variance to inertia (a weighted sum of squared distances; the $\chi^2$ $χ^{2}$ statistic is one instance, phylogenetic diversity another). To compare two tables on the same samples:
- Mantel coefficient — correlate the two distance/dissimilarity matrices $D^X, D^Y$ after vectorizing their off-diagonal entries: $r_m = \frac{\sum_{i\neq j}(d^X_{ij}-\bar d^X)(d^Y_{ij}-\bar d^Y)}{\sqrt{\sum_{i\neq j}(d^X_{ij}-\bar d^X)^2}\;\sqrt{\sum_{i\neq j}(d^Y_{ij}-\bar d^Y)^2}}$ . Distinguished from RV/dCov by the absence of double centering.
- RV coefficient — a correlation-like similarity between two whole tables $A,B$ , $RV(A,B)=\frac{\mathrm{Tr}(A'BAB')}{\sqrt{\mathrm{Tr}(A'A)}\sqrt{\mathrm{Tr}(B'B)}}\in[0,1]$ .
- Canonical Correlation Analysis (CCA, Hotelling). Find linear combinations $U=Xa$ , $V=Yb$ of two variable sets maximizing $\mathrm{cor}(U,V)$ subject to unit variance; the maximizing $\rho_1=\max_{a,b}\mathrm{cor}(Xa,Yb)$ is the first canonical correlation, with higher orthogonal canonical variates $\rho_1\ge\rho_2\ge\cdots$ found stepwise. A PCA-like method whose maximized quantity is cross-table covariance.
- Sparse CCA (sCCA). CCA with an $L_1$ penalty (penaltyx, penaltyz) forcing most coefficients to zero, used as a feature-screening step in the high-dimensional regime (many more features than samples); selected features then feed an ordinary PCA (a triplot: samples, multiple feature domains, and axes together).
- Constrained Correspondence Analysis (CCpnA). CA constrained so the recovered axes lie in the span of supplied explanatory (environmental) variables — a correspondence-analysis analogue of constrained regression. Positions reflect both species/feature signatures and environmental covariates, unlike plain CA/PCA. The chapter deliberately renames it CCpnA to avoid colliding with “CCA” (Canonical Correlation), warning that several R packages overload the name cca.
Named software: cmdscale (classical MDS), metaMDS (NMDS), dudi.coa/ade4, CA/FactoMineR, CCA/ordinate/phyloseq, vegan, PMA (sCCA), ComBat/sva, MatrixCorrelation, ecodist; Bioconductor SummarizedExperiment for keeping assay + covariate data aligned.

3. When It Applies

MDS/PCoA: when objects have meaningful pairwise distances/dissimilarities but no natural coordinates (trees, sequences, images, documents), or when you want a PCA-style map directly from a distance matrix. Use the screeplot’s drop to choose dimensionality.
NMDS: when the scale of the dissimilarities is uncertain or untrustworthy, only their ordering is meaningful, or outliers/large distances would distort a least-squares fit. For the worked color-confusion data, metric and non-metric solutions nearly coincided — the robust version costs little and protects against scale misspecification.
Supplementary points: when most variables are continuous but a few categorical covariates (batch, treatment) must be overlaid to diagnose or adjust for their influence without letting them drive the decomposition.
Correspondence Analysis: for two (or, via MCA, more) categorical variables / contingency tables, after a significant $\chi^2$ test, to locate and interpret which level-pairs drive the dependence. Rows and columns have symmetric status (no response/predictor split).
Local/nonlinear methods (t-SNE, diffusion maps, isomap, LLE): when structure is a low-dimensional manifold or gradient and only local distances are reliable; preferred over global MDS when faraway distances are noisy or the manifold is curved.
Multitable methods: when several assays are measured on the same samples. Choose by goal — Mantel to test whether two distance structures are related; RV for a single scalar table-similarity; CCA/sCCA to extract the correlated linear signal between two variable sets (sCCA when features ≫ samples); CCpnA when you want ordination guided by, but not strictly supervised on, several environmental covariates. CCpnA is “less aggressive than supervised methods” and can use several covariates at once; plain CA has more power to detect association when environmental effects are weak.

4. Assumptions & Validity Conditions

MDS: the input should be (approximately) a distance/dissimilarity matrix. Exact Euclidean reproduction in low dimensions is generally impossible (e.g., curved-surface data into a plane); one accepts an approximate embedding. Negative eigenvalues can appear in cmdscale (unlike PCA) — a normal consequence of non-Euclidean distances, not an error.
Orientation is arbitrary. Because distances are invariant to rotation and reflection, the recovered map is determined only up to rotation/axis-flip; sign and orientation carry no meaning and can vary with trivial data or platform changes.
NMDS: target dimension $k$ must be fixed before fitting; the fit is a stochastic optimization, so results depend on initialization and should be replicated.
CA / $\chi^2$ : rests on the independence (outer-product) model of expected counts; the $\chi^2$ approximation degrades when expected cell counts are very small. Rows and columns are treated symmetrically (no supervised structure).
Classical CCA: assumes $p\le n$ and $q\le n$ with $X,Y$ of full column rank; columns standardized to mean 0, variance 1. These fail in the high-dimensional ( $p,q\gg n$ ) regime, which is exactly why sCCA’s penalty is needed.
Mantel: the entries of a distance matrix are not independent (each object appears in many pairs), so the usual sampling theory for a correlation coefficient does not apply; significance must be obtained by permutation.
Robust methods (NMDS, rank transforms, kernelized/local methods): appropriate precisely when the load-bearing concern is outliers or unreliable large distances; their value is conditional on that being the actual data problem.

5. Failure Modes & Invalidity Patterns

Confounding of effects. If two nuisance factors change together (e.g., protocol and array both differ between days), an ordination shows a clean separation but you cannot attribute it to either cause. Warning sign: groups separate along an axis that aligns with a batch/date variable. Consequence: the apparent effect is uninterpretable. Detection/fix shown explicitly: collect an additional deconfounding batch that breaks the lockstep (one factor held at level A while the other moves to level B); if the new group overlaps the group sharing its true causal level, that factor is the driver. The chapter stresses analyzing data as collected so design flaws are caught in time, not “post mortem.”
Treating covariates as “metadata.” Storing batch/clinical/time information separately from assay data (or encoding a factor as a number that looks continuous) invites lost linkages and category/scale confusion, making later analyses impossible or misleading. The chapter explicitly discourages the “metadata” framing.
Over-interpreting MDS maps. No biplot/loadings exist, so axes have no variable-level meaning; interpretation must come from inspecting extreme points and their contrasts, not from reading axes as in PCA.
Reading orientation/sign as signal. Flipped axes between two analyses (e.g., the sign inversion on a second axis when adding a batch) have no biological relevance; mistaking them for a finding is an error.
t-SNE artifacts. t-SNE is not robust and artificially separates clusters; inter-cluster distances and apparent gaps are not interpretable on a common scale and depend on local point density and on the perplexity/ $\sigma$ tuning. Treating t-SNE separations as true distances, or cluster sizes/gaps as meaningful, is invalid. (The chapter points to a resource on how easily t-SNE is misread.)
Spurious co-variation in high dimensions. With many variables per table, any two highly correlated linear combinations can be found by chance — too many degrees of freedom. A near-perfect canonical/sCCA correlation (e.g., 0.99) does not certify a real shared signal: the scores may be far from bivariate-normal, and other, equally good feature subsets may exist. sCCA minimizes redundancy but offers no guarantee the selected features are the “true” ones. Be wary of the headline correlation.
Mantel mis-significance. Using ordinary correlation p-values for a Mantel coefficient ignores within-matrix dependence and gives wrong significance; the chapter also flags that the Mantel test itself has documented criticisms.
Outlier domination in metric MDS. A single point with large distances to everything can skew the entire configuration; this is the motivation for rank-based/robust and local approaches.

6. Empirical Checks & Calibration

Screeplot of eigenvalues (MDS/CA): a pronounced drop after the first few eigenvalues indicates the effective dimensionality (e.g., a clear break after two axes ⇒ a 2-D embedding suffices; a three-eigenvalue plateau ⇒ genuinely 3-D structure). For CA, the eigenvalue magnitudes indicate how much $\chi^2$ /inertia each axis captures.
Stress curve / “simili-screeplot” for NMDS: rerun the optimization across $k=1,2,3,\dots$ and watch where stress drops steeply; take $k$ after the steep drop. Because each fit is random, run many replicates per $k$ (e.g., on the order of 100) and inspect boxplots of stress to judge both the level and the stability of the solution.
Shepard plot: scatter original dissimilarities against reconstructed distances; a tight monotone relationship signals a faithful embedding for that $k$ .
Cross-method agreement: compare metric vs. non-metric MDS configurations, and MDS under different distances (e.g., $\ell_1$ vs. $L_2$ ) — near-identical maps and near-identical eigenvalue profiles are reassurance that the recovered structure is real and not an artifact of one metric; large disagreement flags fragility.
Sensitivity to tuning: vary the kernel $\sigma$ (and t-SNE perplexity/dimension) and check whether structure persists; structure that appears only at one setting is suspect.
Permutation testing for table association: the Mantel coefficient’s null distribution is obtained by permuting one matrix’s labels and recomputing the statistic (the only valid significance route given dependent distances); RV.test similarly assesses table-level association before committing to a multitable model.
Ground-truth / known-structure checks: validating an embedding against a known configuration when one exists (the geography example reconstructs true relative city positions up to rotation; ordering recovered by seriation can be checked against known endpoints) demonstrates the method recovers truth and exposes its rotation/reflection ambiguity.
Batch-removal verification: after adjusting (group-mean recentering, or ComBat/empirical-Bayes), redo the ordination and confirm the previously separated batch groups now overlap near the origin while the biological clustering remains.
Rank/threshold transforms as robustness controls: replacing values by ranks, and tying low (noise-level) values at a threshold, prevents meaningless large rank jumps among absent/low-abundance features from creating spurious signal.

7. Interpretation, Guidance & Trade-offs

What a map does and does not license. Proximity in an MDS/NMDS map means similar (small distance); axes have no intrinsic meaning, and sign/orientation are arbitrary. In CA, proximity of a row level and a column level means strong co-occurrence (barycentric reading), and the biplot shows positive vs. negative departures from independence (e.g., one phenotype pairing occurs more than expected, another less). t-SNE clusters are real groupings, but between-cluster distances and gap sizes are not quantitatively meaningful.
CA as the honest sequel to $\chi^2$ . A significant $\chi^2$ only says “dependent”; CA localizes the dependence to specific level-pairs and quantifies what fraction of the total $\chi^2$ /inertia the first axes explain — report that percentage as the analogue of “variance explained.”
Robust vs. exact trade-off. Metric MDS uses distance values (more information, less robust); NMDS uses only ranks (robust to outliers and scale misspecification, but loses the metric and the per-axis variance accounting). When both agree, prefer the simpler interpretation; when they disagree, trust the robust one if outliers are plausible.
Global vs. local trade-off. Global methods (MDS) faithfully represent large distances but can be dominated by them; local/kernel methods (t-SNE, diffusion maps, isomap) sacrifice global geometry to render local/manifold structure and gradients — choose by whether the scientific target is overall configuration or a trajectory.
Keep data and covariates together. Treat batch/time/protocol/clinical covariates as first-class data tightly linked to the assays (e.g., in a SummarizedExperiment); this both enables correct filtering and makes batch effects detectable as supplementary points.
Report for trustworthiness: the distance/dissimilarity used, the chosen dimensionality and how it was selected (scree/stress), replicate stability for stochastic methods, tuning parameters (kernel $\sigma$ , perplexity, sparsity penalties) and the number of features selected, and any batch-adjustment applied. For sCCA, hold in mind and disclose that the selected feature set is one of possibly many and the reported canonical correlation is optimistic.
Sparse/regularized defaults: in the high-dimensional regime prefer the regularized ( $L_1$ -penalized) CCA over classical CCA, and tune penalties toward more sparsity than defaults to keep the result interpretable.

8. Connections & Key Terms

Builds on: PCA and SVD as best low-rank approximation (the prior continuous-only ordination); clustering as discovery of a hidden categorical latent factor; the $\chi^2$ test of independence for contingency tables; correlation/covariance for paired continuous variables; rank-based/nonparametric robustness. Sets up: trajectory/pseudotime and developmental-lineage analysis via gradients and manifolds; networks and trees (graph/tree distances as ordination input, phylogenetic diversity as inertia); multi-omics data integration via multitable correlation; multiple correspondence analysis (MCA) for many categorical (and binned continuous) variables; regularization/overfitting concerns in high-dimensional supervised learning.

Key terms:

Ordination — building a low-dimensional map that reveals a hidden continuous ordering/gradient (the continuous analogue of clustering).
MDS / PCoA — recovering point coordinates from a distance matrix by double-centering and eigendecomposition; PCA’s distance-space cousin.
NMDS — rank-only, robust MDS minimizing stress.
Stress — standardized residual sum of squares between input dissimilarities (monotonically transformed) and map distances.
Breakdown point — fraction of data corruption an estimator tolerates before failing ( $1/2$ median, $1/n$ mean); a robustness measure.
Centering matrix $H$ — $I-\tfrac1n\mathbf{1}\mathbf{1}^t$ ; subtracts column/row means to remove origin dependence.
Horseshoe / arch — curved point pattern signaling an underlying one-dimensional gradient or chronology; basis of seriation.
Seriation — ordering objects along a latent sequence from presence/absence or abundance patterns.
Contingency table — cross-tabulated co-occurrence counts of two categorical variables.
$\chi^2$ distance / statistic — weighted measure of a table’s departure from the independence (marginal outer-product) model; CA’s analogue of variance.
Correspondence Analysis (CA) — weighted SVD/MDS of $\chi^2$ distances yielding a symmetric biplot of row and column levels; barycentric.
Supplementary point — a group mean (or extra observation) projected onto an existing map without entering the decomposition; used to overlay covariates and spot batch effects.
Batch effect / confounding — nuisance technical variation; confounded when two factors vary together so their effects cannot be separated without a deconfounding batch.
Inertia — weighted sum of squared distances; the multivariate generalization of variance.
Mantel coefficient — correlation between two distance matrices’ vectorized entries; significance via permutation (no double centering).
RV coefficient — $[0,1]$ correlation-like similarity between two whole data tables.
Canonical Correlation Analysis (CCA) — linear combinations of two variable sets with maximal mutual correlation; sCCA adds an $L_1$ penalty for feature selection when features ≫ samples.
Constrained Correspondence Analysis (CCpnA) — CA forced into the span of explanatory environmental variables; ordination guided by covariates, gentler than full supervision.
t-SNE / diffusion map / isomap / LLE — local, nonlinear embedding methods that prioritize local distances and reveal manifolds/trajectories; t-SNE inter-cluster distances are not metrically interpretable.