A dataset begins with twenty variables. Many say nearly the same thing. Several move together. A few pull in opposite directions. The analyst wants a smaller map, but there are two very different ambitions hidden inside that request.
One ambition is geometric: find new axes that capture as much observed variation as possible with fewer coordinates. The other is explanatory: propose that correlations among observed variables arise partly because a smaller number of unobserved factors influence them.
Principal component analysis, or PCA, constructs orthogonal linear combinations of observed variables ordered by the variance they capture. Factor analysis posits a latent-variable model in which observed variables share variation through underlying factors plus variable-specific uniqueness. The methods are related because both begin from multivariate covariance structure. They are not interchangeable, and calling both “dimension reduction” can conceal their different questions.
This article owns the reader job of understanding that distinction. Correlation, Covariance and Association owns the pairwise foundation. Construct Validity and Measurement Models owns the broader problem of whether a score supports an intended interpretation. Measurement Invariance and Fair Comparisons owns whether a measurement structure behaves comparably across groups or time.
Reading route: build the covariance matrix, then learn PCA geometry, scaling choices, component retention, the factor model, rotation and indeterminacy, PCA versus factor analysis, and finally responsible interpretation and validation.
Multivariate analysis begins with a matrix of shared variation
With p numerical variables, the variance-covariance matrix places each variable’s variance on the diagonal and each pairwise covariance off the diagonal. The correlation matrix standardises the variables so the diagonal entries are one and the off-diagonal entries are correlations.
These matrices are more than tables. They encode the geometry of how the observed cloud spreads through p-dimensional space. If several variables rise and fall together, much of the cloud may lie near a lower-dimensional direction. PCA formalises that intuition.
Penn State’s STAT 505 Applied Multivariate Statistical Analysis course treats covariance, correlation, PCA and factor analysis as connected stages of multivariate reasoning. Its current course materials were available and checked for this edition on 5 September 2026.
PCA rotates the coordinate system to directions of greatest variance
Imagine a two-variable scatterplot shaped like a long tilted ellipse. The original horizontal and vertical axes may be arbitrary measurement choices. The longest direction through the cloud captures the greatest variation. PCA chooses that direction as the first principal component.
The second principal component is constrained to be orthogonal to the first and captures the greatest remaining variance. In p dimensions, PCA continues until p components have been defined. If the first few capture most of the meaningful variation, they can provide a lower-dimensional representation.
Mathematically, the directions are eigenvectors of the covariance or correlation matrix, and the corresponding eigenvalues give the variance of the component scores. Penn State’s PCA lesson develops this relationship through spectral decomposition and interpretable linear combinations.
A principal component is a linear combination, not a newly observed trait
If X₁,…,Xₚ are centred variables, a principal component has the form Z = a₁X₁ + … + aₚXₚ. The coefficients are chosen to optimise variance subject to normalisation and orthogonality constraints.
The component score for one observation is calculated by inserting that observation’s values into the linear combination. The component therefore exists because the analyst constructed a coordinate from the observed variables. It is not automatically a hidden psychological, social or physical cause.
Interpretation can still be useful. If several education-access variables have large positive coefficients on one component, an analyst may describe the component as an access-related pattern. That label is an interpretation of a variance direction, not proof that a single latent entity called “access” generated the data.
Eigenvalues tell us how much variance each component carries
The eigenvalue associated with a component equals the variance of that component’s scores. The sum of all eigenvalues equals the total variance represented in the matrix. This makes the proportion of explained variance straightforward: divide a component’s eigenvalue by the total.
If standardised variables are analysed through a correlation matrix, total variance equals the number of variables because each has variance one. An eigenvalue of 3 then means that component carries as much standardised variance as three original variables combined.
Explained variance is descriptive, not automatically meaningful variance. A component can capture a great deal of variation caused by batch effects, measurement scale, calendar timing or another nuisance source. Variance maximisation does not know the analyst’s scientific question.
Loadings describe how variables relate to components
The word loading is used somewhat differently across software and methods, so definitions should be checked. In PCA, loadings often refer to coefficients or to correlations between variables and component scores. Both help interpret which observed variables contribute strongly to a component.
Large magnitude matters more than sign alone. The sign of an eigenvector can be flipped without changing the PCA solution: a component with all coefficients multiplied by −1 represents the same axis pointing in the opposite direction. Therefore “positive” and “negative” component labels require a declared orientation.
This sign indeterminacy is a mathematical symmetry, not a defect. What matters is the relative pattern of coefficients and scores.
Covariance-matrix PCA and correlation-matrix PCA can answer different questions
Suppose one variable is measured in thousands and another in fractions. A covariance-matrix PCA gives more influence to variables with larger numerical variance. Sometimes that is exactly right because their absolute scale carries meaning. Sometimes it makes the result largely a unit-choice artefact.
Standardising each variable to mean zero and variance one before PCA is equivalent to using the correlation matrix. This puts variables on comparable variance scales. It is common when units differ substantially or no variable should dominate merely because of its measurement scale.
Standardisation is not automatically neutral. A variable with tiny but highly reliable variation can be elevated to equal variance with a noisy variable. A naturally dominant physical quantity can be artificially reduced to the same variance as a minor feature. The choice belongs to the reader job and measurement theory.
A worked two-variable PCA
Consider two centred, standardised variables with correlation 0.8. Their correlation matrix is [[1, 0.8],[0.8,1]]. Its eigenvalues are 1.8 and 0.2. The first eigenvector is proportional to (1,1), and the second to (1,−1).
The first component is therefore essentially the average direction: both variables high together or low together. It explains 1.8/2 = 90% of the standardised variance. The second component is a contrast: one variable high relative to the other. It explains the remaining 10%.
If the analyst retains only the first component, the dataset is compressed from two numbers to one score while discarding the contrast dimension. Whether that loss is acceptable depends on what the second dimension means for the task.
PCA can reconstruct an approximation to the original data
Keeping all components is simply a rotation of the centred coordinate system; no information is lost apart from numerical precision. Keeping only the first k components projects observations onto a lower-dimensional subspace.
The discarded components quantify the directions of omitted variation. Reconstruction error provides one way to assess the cost of compression. A low reconstruction error can be useful for storage, visualisation or denoising, but low error does not guarantee preservation of the particular feature needed for a downstream decision.
How many components should be retained?
There is no universally correct number. Common tools include cumulative explained variance, scree plots, comparison with noise or random-data benchmarks, cross-validation for a downstream task and subject-matter interpretability.
A scree plot graphs eigenvalues against component number. Analysts often look for an elbow where large eigenvalues give way to a flatter tail. The elbow can be ambiguous. The graph is evidence, not an oracle.
The “eigenvalue greater than one” rule sometimes used with correlation-matrix PCA is a heuristic based on retaining components that explain more variance than one standardised original variable. It should not be treated as a theorem guaranteeing the right dimensionality.
Parallel analysis compares observed eigenvalues with those expected from suitably generated random data and can provide a stronger empirical benchmark. Even then, the retained structure must make sense for the intended interpretation or prediction job.
PCA scores can be useful predictors, but prediction changes the evaluation criterion
Principal components are chosen to explain predictor variance, not necessarily outcome-relevant variance. A low-variance direction could be highly predictive of an outcome and be discarded by unsupervised PCA.
If PCA is used inside a predictive workflow, the transformation must be fitted only on the training data within each validation split. Fitting PCA once on the full dataset before cross-validation leaks information from held-out observations into the feature construction.
The companion Model Selection, Regularisation and Cross-Validation article owns that pipeline rule in depth.
Uncorrelated principal components are not necessarily statistically independent
PCA constructs components with zero covariance in the analysed data geometry. Zero covariance does not generally imply independence. If the joint distribution is multivariate normal, uncorrelated linear combinations have stronger independence properties. Outside such conditions, higher-order dependence can remain.
This distinction matters when PCA is used as though it had separated independent sources. That stronger task belongs to methods such as independent component analysis under different assumptions.
Factor analysis begins with a latent-variable model
A common factor model represents an observed variable vector X approximately as ΛF + ε, after any mean structure is handled. F is a smaller vector of latent common factors, Λ is a loading matrix, and ε contains variable-specific uniqueness or error.
The model says that correlations among observed variables can be explained partly by their shared dependence on common factors. This is more than a coordinate rotation. It is a probabilistic or structural proposal about shared and unique sources of variation.
Penn State’s STAT 505 course includes a dedicated Factor Analysis lesson covering the factor model, extraction, loadings and rotation. The course route is useful because it places factor analysis after PCA while keeping their purposes distinct.
Communality and uniqueness split the variance of an observed variable
Under a standard factor model, the variance of an observed variable can be decomposed into common variance explained by the factors and unique variance specific to that variable. The common portion is its communality; the remainder is uniqueness.
A high communality means the retained common-factor structure accounts for much of that variable’s variance under the fitted model. A low communality means the variable is poorly represented by the common factors or contains substantial specific/error variation.
Calling uniqueness “measurement error” is too strong unless the model and study justify that interpretation. Unique variance can include genuine variable-specific content as well as noise.
Factor extraction is model fitting, not merely matrix decomposition
Different extraction methods make different assumptions and optimise different criteria. Maximum-likelihood factor analysis, for example, brings a likelihood model and distributional conditions that allow inferential tests under those conditions. Principal-axis-type methods target common variance differently.
Software menus can make extraction look like a style choice. It is not. The method determines what is estimated, which assumptions support inference and how uncertainty should be assessed.
Rotation changes the factor representation to improve interpretability
Once several factors are retained, the initial factor axes may not have a simple interpretation. Rotation searches for an alternative orientation of the retained factor space that produces a clearer loading pattern while preserving the model’s essential representation according to the rotation class.
Orthogonal rotations keep factors at right angles and therefore uncorrelated in the chosen factor-coordinate representation. Oblique rotations allow factors to correlate. If the constructs are expected to be related, forcing orthogonality for convenience can distort interpretation.
Rotation demonstrates an important truth: latent-factor coordinates are not uniquely determined by covariance fit alone. Multiple rotated solutions can reproduce the same common covariance structure while telling the story in different axes.
Factor labels are hypotheses, not discoveries printed by software
Suppose items about planning, persistence and self-monitoring load on one factor. Naming it “self-regulation” may be reasonable if theory and item content support the interpretation. The loading pattern alone does not prove the factor is a natural psychological entity or that it causes the responses.
Labels should be tested against external evidence, alternative models and intended score use. A factor that looks interpretable in one sample can split, merge or rotate differently in another population.
Factor scores are estimates of latent positions, not direct measurements
After fitting a factor model, analysts may compute factor scores for each observation. Several scoring methods exist because latent factor values are not directly observed and are not uniquely recoverable without additional assumptions.
Using factor scores as though they were error-free measured variables can understate uncertainty. If the scores enter a later regression or classification, their estimation process and reliability can matter.
PCA and factor analysis answer different questions
| Dimension | Principal component analysis | Common factor analysis |
|---|---|---|
| Primary job | Represent observed variance with orthogonal linear combinations | Model shared covariance through latent common factors |
| Variance treated | Total observed variance | Common variance separated from uniqueness |
| Core object | Eigenstructure of covariance/correlation matrix | Latent-variable covariance model |
| Coordinates | Components are determined up to sign when eigenvalues are distinct | Factors have rotational indeterminacy without additional constraints |
| Interpretation | Variance directions may be labelled descriptively | Latent constructs require substantive validation |
| Typical use | Compression, visualisation, preprocessing, denoising | Measurement structure, latent dimensions, covariance explanation |
The methods can produce superficially similar loading tables, especially when data have strong simple structure. That visual resemblance does not erase the different estimands and assumptions.
PCA is not exploratory factor analysis with a different button
In PCA, every observed variance contribution is part of the component decomposition. In common factor analysis, the model distinguishes common covariance from uniqueness. Treating PCA components as latent factors can exaggerate the interpretation because PCA was not designed to separate measurement-specific variance from common variance.
Conversely, using a complicated factor model when the only job is numerical compression can add assumptions and interpretive burden that the task does not need. The method should follow the reader job.
Exploratory factor analysis and confirmatory factor analysis are different stages
Exploratory factor analysis allows the loading pattern to be learned with relatively few fixed structural restrictions, subject to the chosen extraction and rotation. Confirmatory factor analysis begins with a more explicit measurement model specifying which variables load on which factors and which parameters are constrained.
A model that was explored and modified repeatedly on one dataset should not then be described as independently confirmed by that same dataset. Holdout samples, new data or preregistered confirmatory analyses provide stronger evidence for stability.
Preregistration and Registered Reports owns the broader distinction between planned confirmation and data-driven exploration.
Sample size cannot be reduced to one universal observations-per-variable rule
Rules such as five or ten observations per variable are common heuristics, not guarantees. Stability depends on factor strength, communalities, number of indicators per factor, model complexity, distribution, missingness and estimation method.
A strong simple structure can sometimes be recovered with fewer observations than a weak complex structure. A huge sample can precisely estimate a poorly conceptualised model. Planning should consider both statistical stability and the substantive adequacy of the indicators.
Missing data can change covariance structure before either method begins
PCA and factor analysis depend on estimates of multivariate relationships. If missingness changes which records contribute to which covariances, the matrix may no longer represent one coherent target population. Pairwise deletion can even produce a matrix with problematic mathematical properties.
Model-based missing-data methods can be useful under defensible assumptions, but filling blanks mechanically before multivariate analysis can create artificial structure. How Missing Data Analysis Works owns that decision.
Outliers and mixtures can create components or factors that are really regime indicators
A small cluster of extreme observations can rotate principal directions. Two subpopulations with different means can generate a strong first component that mainly separates the groups. Factor analysis may then fit covariance generated by an unmodelled mixture.
Inspect distributions, scatterplots and group structure before interpreting a loading table. A component called “performance” may actually be “site A versus site B” if the sites were measured under different conditions.
Ordinal items need measurement-aware treatment
Likert-type or ordered categorical items are often analysed using Pearson correlations and continuous latent models. That can be an acceptable approximation in some settings and poor in others. Alternatives use ordinal models and polychoric-type correlations under additional assumptions.
The choice should reflect category count, response distributions, model purpose and intended interpretation. Treating every coded number as interval-scale measurement merely because software accepts it is not a measurement argument.
A factor structure can change across groups even when item wording is identical
Suppose the same questionnaire is used across age groups. A two-factor structure in one group may become three factors in another, or loadings and intercepts may differ. Comparing raw or factor scores as though the measurement meaning were identical can then be misleading.
This is exactly where Measurement Invariance and Fair Comparisons becomes the canonical next owner.
High explained variance does not establish construct validity
A first component explaining 70% of variance may look impressive. But if the intended construct is “critical thinking”, the component needs evidence that its indicators and score meaning really support that interpretation. It could instead reflect reading speed, test-taking strategy or a general response style.
Statistical coherence is one source of evidence. Construct validity requires a wider argument connecting content, internal structure, relationships with other variables, response processes and consequences to the intended use.
A good factor model does not prove a causal latent mechanism
Several causal structures can produce the same covariance pattern. A common factor may cause responses, responses may influence one another, an omitted process may create shared variation, or the factor model may simply be a useful statistical representation.
Causal interpretation requires stronger design and theory than covariance fit. How Causal Inference Works owns that boundary.
Stability across samples is part of the evidence
A loading pattern can look clean because one sample happened to align favourably. Bootstrap analyses, split-sample checks or independent replication can reveal how stable components, retained dimensions and loadings are.
PCA poses a special matching problem across resamples because component signs can flip and near-equal eigenvalues can allow directions to rotate substantially while representing nearly the same subspace. Factor analysis adds rotational choices. Stability should therefore be assessed at the level that is actually identified: eigenvalues, subspaces, loading patterns, communalities or predictive performance as appropriate.
How Reproducibility and Replication Work provides the wider research route.
Cross-validation changes what “good dimension reduction” means
If the goal is reconstruction, evaluate held-out reconstruction. If the goal is prediction, evaluate downstream predictive performance with the dimension-reduction step refitted inside each training fold. If the goal is measurement, cross-validation alone is insufficient because interpretive validity and invariance also matter.
The metric must follow the use. A factor solution that predicts well may be difficult to interpret. A beautifully interpretable factor structure may add no predictive value. Neither property automatically outranks the other outside a defined reader job.
When PCA is the right acquisition
PCA is especially useful when many continuous variables are redundant and the immediate goal is compact representation, visualisation, noise reduction or preprocessing. It is attractive because the optimisation is explicit and the component axes are uniquely ordered by variance when eigenvalues are distinct.
It is less appropriate when the analyst needs a theory of latent common causes, categorical measurement structure or direct interpretability of original variables without transformation.
When factor analysis is the right acquisition
Factor analysis is useful when the substantive question concerns shared latent dimensions believed to generate covariance among observed indicators. It belongs naturally in measurement development and construct research when supported by theory and validation.
It is not automatically useful just because a dataset has many columns. If those columns are operational measurements with no plausible common-factor interpretation, PCA, clustering, regression or a domain-specific physical model may be better suited.
A practical audit before interpreting components or factors
- Define the unit of analysis and target population.
- Inspect variable meanings, units, ranges, missingness and measurement quality.
- Examine distributions, scatterplots and the covariance/correlation matrix.
- Decide whether the job is compression, prediction or latent measurement.
- For PCA, justify covariance versus correlation scaling.
- For factor analysis, state the latent model, extraction method and rotation strategy.
- Use more than one retention diagnostic when dimensionality is consequential.
- Check outliers, mixtures and dependence structures that can dominate covariance.
- Assess stability across samples or resamples.
- Keep component/factor labels provisional until external and substantive evidence supports them.
- If comparing groups or time, test measurement comparability rather than assuming it.
- Do not infer causal latent mechanisms from covariance fit alone.
A learner’s mental model: compression and explanation are different ambitions
PCA asks, “Can I describe this observed cloud with fewer perpendicular directions?” Factor analysis asks, “Can a smaller set of latent common variables plausibly explain why these observed variables share covariance?”
The first is a geometry of representation. The second is a model of shared structure. They meet at the covariance matrix and then walk different paths.
That distinction prevents two opposite errors: inventing hidden constructs from a convenient PCA loading table, and burdening a simple compression problem with a latent-variable theory it never needed.
Sources and further reading
Source pages were checked for this edition on 5 September 2026. The numerical examples are explanatory constructions. Applying PCA or factor analysis to consequential measurement decisions requires study-specific diagnostics and subject-matter expertise.
- Penn State STAT 505 — Applied Multivariate Statistical Analysis.
- Penn State STAT 505 — Principal Components Analysis.
- Penn State STAT 505 — Covariance and Correlation Foundations.
- Penn State Statistics Online — STAT 505 course description and current course topics.
Continue through eduKate: read Correlation, Covariance and Association, Construct Validity and Measurement Models, Measurement Invariance and Fair Comparisons, Model Selection, Regularisation and Cross-Validation, and return through the Research Collections Directory.