Singular Value Decomposition
SVD is the Swiss Army knife of linear algebra. Every matrix has one. Every data scientist needs one. Implement SVD via power iteration and explain the geometric meaning of U, Sigma, and V^T. Apply truncated SVD for image compression and measure the compression ratio vs reconstruction error. Compute the Moore-Penrose pseudoinverse via SVD to solve overdetermined least-squares systems. Connect SVD to PCA, recommendation systems (latent factors), and Latent Semantic Analysis in NLP. You have a 1000x2000 matrix. Maybe it is user-movie ratings. Maybe it is a document-term frequency table. Maybe it is the pixel values of an image. You need to compress it, denoise it, find hidden structure in it, or solve a least-squares system with it. Eigendecomposition only works on square matrices. Even then, it requires the matrix to have a full set of linearly independent eigenvectors. SVD works on any matrix. Any shape. Any rank. No conditions. It decomposes the matrix into three factors that reveal the geometry of what the matrix does to space. It is the most general and most useful factorization in all of linear algebra. Every matrix, regardless of shape, performs three operations in sequence: rotate, scale, rotate. SVD makes this decomposition explicit. Given any matrix A, SVD factors it into: V^T rotates vectors in the input space (n-dimensional). Sigma scales along each axis…
Singular Value Decomposition: SVD is the Swiss Army knife of linear algebra. Every matrix has one. Every data scientist needs one.
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.