Eigenvalues and SVD Overview

An eigenvector of a square matrix A is a non-zero vector v such that A @ v = lambda * v — multiplying by A only scales v, it does not change its direction. The scalar lambda is called the eigenvalue corresponding to v. Eigenvalues reveal the intrinsic 'stretching factors' of a linear transformation: an eigenvalue of 2 means the matrix doubles lengths in the eigenvector direction; a negative eigenvalue flips direction.

np.linalg.eig(A) returns a tuple (eigenvalues, eigenvectors). The eigenvalues are a 1-D array; the eigenvectors are a 2-D array where each column is an eigenvector. For real symmetric matrices (like covariance matrices), the eigenvalues are always real and the eigenvectors are orthogonal — use np.linalg.eigh(A) for symmetric matrices as it is faster and guaranteed to return real results.

Eigenvalues encode important matrix properties. The determinant equals the product of all eigenvalues: det(A) = product(eigenvalues). The trace (sum of diagonal elements) equals the sum of eigenvalues: trace(A) = sum(eigenvalues). A matrix is positive definite (all eigenvalues > 0) only if all eigenvalues are positive — a crucial property for valid covariance matrices and convex optimisation problems.

Singular Value Decomposition (SVD) decomposes any matrix A (not just square ones) as A = U @ S @ V.T where U and V are orthogonal matrices and S is diagonal with non-negative singular values on the diagonal. SVD is the most general and numerically stable matrix factorisation. It underpins PCA, image compression, recommender systems, and the pseudo-inverse computation. np.linalg.svd(A) returns U, s (1-D singular values), and Vh (V transposed).

The singular values (diagonal of S) are always non-negative and conventionally listed in descending order. The number of non-zero singular values equals the rank of the matrix. Near-zero singular values indicate near-linear-dependence among rows or columns. The largest singular value gives the spectral norm of the matrix, and the ratio of largest to smallest non-zero singular value is the condition number used to measure numerical stability.

Principal Component Analysis (PCA) can be implemented directly via SVD. After centering your data matrix X (subtracting column means), the right singular vectors in Vh are the principal components, and the singular values squared (divided by n-1) are the variances explained. Sklearn's PCA uses exactly this approach. Understanding this connection lets you implement or customise PCA from scratch for dimensionality reduction.

SVD enables low-rank matrix approximation: keep only the top k singular values and vectors, and reconstruct an approximation of the original matrix. This is the basis of image compression and collaborative filtering for recommendations. The truncated SVD U[:, :k] @ np.diag(s[:k]) @ Vh[:k, :] gives the best rank-k approximation in the least-squares sense (Eckart-Young theorem).

Use eigendecomposition (np.linalg.eig or eigh) when you have a square symmetric matrix and want to understand its principal axes — for example, the covariance matrix in PCA or Markov transition matrices. Use SVD (np.linalg.svd) when your matrix is rectangular or when you need maximum numerical stability. SVD always exists; eigendecomposition can produce complex numbers for non-symmetric matrices.

The spectral theorem states that every real symmetric matrix A can be decomposed as A = Q @ diag(eigenvalues) @ Q.T where Q is orthogonal (Q.T = Q⁻¹). This means symmetric matrices are always diagonalisable with real eigenvalues and orthogonal eigenvectors. Covariance matrices, kernel matrices in SVMs, and the Hessian in optimisation are all symmetric, making this theorem universally useful in machine learning theory.

When you only need the largest eigenvalue and its eigenvector, power iteration is far more efficient than computing all eigenvalues. Start with a random vector, repeatedly multiply by A and normalise, and it converges to the dominant eigenvector. This is how Google's original PageRank algorithm worked. NumPy makes each iteration a single matrix-vector multiply with @.

In a real data pipeline, SVD is used for noise reduction and data compression. After fitting SVD on training data, you keep only the top k components that capture 95% of the variance. This reduces the dimensionality of new data before feeding it to a classifier or regressor, speeding up training and often improving generalisation by removing noisy dimensions. Always fit SVD on training data only and apply the same transform to test data.

Test your understanding of Data Analysis concepts from this lesson.

In this lesson you learned: np.linalg.eig()/eigh() compute eigenvalues and eigenvectors that reveal intrinsic stretching directions of a matrix, np.linalg.svd() decomposes any matrix into U, singular values, and Vh enabling PCA and low-rank approximation, and singular values quantify the variance captured by each component and determine the matrix rank. Next up we tackle handling large datasets by streaming CSV files in chunks.