Linear Algebra & Geometry: Vector Spaces, Matrices & Geometric Transformations

📖 What is Linear Algebra? The Mathematics of Vector Spaces

Linear algebra is the branch of mathematics that studies vector spaces and linear transformations. It provides the language for describing multidimensional space, systems of equations, and transformations that preserve the structure of vector spaces. From solving systems of equations to powering recommendation algorithms, linear algebra is ubiquitous in modern science and technology.

The Central Idea: Linearity

A transformation T is linear if it satisfies two properties: additivity (T(u+v) = T(u) + T(v)) and homogeneity (T(cv) = c T(v)). This seemingly simple structure yields profound consequences. Linear transformations preserve the operations of vector addition and scalar multiplication, which means they map lines to lines and preserve parallelism. This property makes linear algebra tractable and powerful.

Why Linear Algebra Matters

Linear algebra is the hidden engine behind countless modern technologies:

• Machine Learning: Neural networks, support vector machines, and principal component analysis are built on linear algebra
• Computer Graphics: 3D rendering, game physics, and animation rely on matrix transformations
• Quantum Mechanics: The state of a quantum system is a vector in Hilbert space; observables are linear operators
• Data Science: Dimensionality reduction, recommendation systems, and regression models use linear algebra
• Engineering: Structural analysis, control systems, and signal processing are fundamentally linear

📦 Vector Spaces: The Foundation of Linear Algebra

Definition of a Vector Space

A vector space V over a field F (typically ℝ or ℂ) is a set equipped with two operations: vector addition and scalar multiplication, satisfying eight axioms. These axioms ensure that vector spaces behave predictably and include familiar examples like ℝⁿ, spaces of functions, and polynomial spaces.

Subspaces, Span, and Basis

A subspace is a subset of a vector space that is itself a vector space (closed under addition and scalar multiplication). The span of a set of vectors is the set of all linear combinations—the smallest subspace containing those vectors. A basis is a linearly independent spanning set; it provides a coordinate system for the vector space. The number of vectors in any basis is the dimension of the space.

For ℝⁿ, the standard basis vectors e₁ = (1,0,...,0), e₂ = (0,1,0,...,0), etc., form a basis. But infinitely many other bases exist, each providing a different coordinate system. The choice of basis is crucial: the right basis can transform a complex problem into a simple one. This is why eigenvalue problems—finding bases that diagonalize transformations—are so important.

Linear Independence and Dimension

A set of vectors is linearly independent if no vector can be written as a linear combination of the others. Formally, c₁v₁ + c₂v₂ + ... + cₖvₖ = 0 implies all c_i = 0. Linear dependence means there is redundancy—some vector is unnecessary for spanning the space. The dimension theorem states that all bases of a finite-dimensional vector space have the same size, establishing the concept of dimension.

🔢 Matrices and Linear Transformations: The Bridge Between Algebra and Geometry

Matrices as Linear Maps

Every linear transformation between finite-dimensional vector spaces can be represented by a matrix, given choices of bases. If T: ℝⁿ → ℝᵐ is linear, there exists an m×n matrix A such that T(v) = Av for all v. The columns of A are the images of the standard basis vectors: A = [T(e₁) T(e₂) ... T(eₙ)].

Fundamental Subspaces

Associated with any m×n matrix A are four fundamental subspaces:

• Column Space (Col A): Span of columns of A; the set of all possible outputs Ax
• Nullspace (Null A): Solutions to Ax = 0; vectors mapped to zero
• Row Space (Row A): Span of rows of A; orthogonal complement of nullspace
• Left Nullspace: Solutions to yᵀA = 0; orthogonal complement of column space

The Rank Theorem states that rank(A) + nullity(A) = n, where rank is the dimension of the column space and nullity is the dimension of the nullspace. This fundamental relationship captures the essence of linear transformations: information is either preserved (in the column space) or lost (in the nullspace).

Determinants and Volume

The determinant det(A) measures how a linear transformation scales volumes. For a 2×2 matrix, |det(A)| is the factor by which area changes; for 3×3, volume scaling. If det(A) = 0, the transformation collapses space onto a lower-dimensional subspace (non-invertible). If det(A) ≠ 0, the transformation is invertible, preserving dimension. The sign of the determinant indicates whether orientation is preserved (positive) or reversed (negative).

⚡ Eigenvalues and Eigenvectors: The Heart of Linear Algebra

Definition and Geometric Meaning

For a square matrix A, an eigenvector is a nonzero vector v such that Av = λv for some scalar λ (the eigenvalue). Geometrically, eigenvectors are directions that are invariant under the transformation—they may be stretched or compressed, but not rotated out of their original direction.

Eigenvalues are found by solving the characteristic equation det(A - λI) = 0, an nth-degree polynomial. The set of all eigenvalues is the spectrum of A.

Diagonalization and Spectral Theorem

If a matrix has n linearly independent eigenvectors, it can be diagonalized: A = PDP⁻¹, where D is diagonal with eigenvalues on the diagonal, and P's columns are eigenvectors. Diagonalization simplifies computation: Aᵏ = PDᵏP⁻¹, making powers of matrices trivial.

The Spectral Theorem is one of the most important results: any real symmetric matrix (Aᵀ = A) can be diagonalized by an orthogonal matrix. This means there exists an orthonormal basis of eigenvectors, and A can be written as A = QΛQᵀ, where Q is orthogonal (Qᵀ = Q⁻¹) and Λ is diagonal. Symmetric matrices appear everywhere in applications: covariance matrices, adjacency matrices of undirected graphs, and Hessian matrices in optimization.

📐 Inner Product Spaces: Geometry in Vector Spaces

The Dot Product and Beyond

An inner product generalizes the dot product to abstract vector spaces, providing notions of length, angle, and orthogonality. For ℝⁿ, the standard inner product is ⟨x,y⟩ = xᵀy = Σ x_i y_i. In function spaces, inner products involve integrals: ⟨f,g⟩ = ∫ f(x)g(x) dx.

Orthogonality and Projections

Vectors are orthogonal if ⟨u,v⟩ = 0. Orthogonal sets are automatically linearly independent. An orthonormal basis—a basis of mutually orthogonal unit vectors—simplifies coordinate representation: the coefficient of each basis vector is simply the inner product with that vector.

The Gram-Schmidt process converts any basis into an orthonormal basis. It works by iteratively subtracting projections onto previously constructed vectors, then normalizing. Orthonormal bases are essential for the QR factorization (A = QR, with Q orthogonal and R upper triangular).

Least Squares and Best Approximations

Perhaps the most practical application: given data points, find the line (or plane) of best fit. The least squares solution to Ax = b (when inconsistent) minimizes ‖Ax - b‖². The solution is x̂ = (AᵀA)⁻¹Aᵀb, representing the orthogonal projection of b onto the column space of A. This is the mathematical foundation of linear regression.

📉 Singular Value Decomposition: The Ultimate Matrix Factorization

What is SVD?

The Singular Value Decomposition (SVD) factorizes any m×n matrix A into A = UΣVᵀ, where:

• U is m×m orthogonal (columns are left singular vectors)
• Σ is m×n diagonal with nonnegative singular values σ₁ ≥ σ₂ ≥ ... ≥ σ_r > 0
• V is n×n orthogonal (columns are right singular vectors)

SVD exists for every matrix—no restrictions! This universality makes it the most powerful matrix decomposition.

Applications of SVD

• Principal Component Analysis: The right singular vectors are the principal components; singular values indicate variance explained
• Low-Rank Approximation: Truncating SVD gives the best rank-k approximation (Eckart-Young theorem)
• Recommendation Systems: Netflix Prize winners used SVD to predict user ratings
• Image Compression: Keep largest singular values, discard small ones—massive compression with minimal visual loss
• Pseudoinverse: A⁺ = VΣ⁺Uᵀ solves least squares problems robustly
• Latent Semantic Analysis: Extracting semantic structure from text documents

🔄 Geometric Transformations in 2D and 3D

Basic Transformations

All these transformations are linear (or affine) and can be represented by matrices:

• Rotation: R(θ) = [[cos θ, -sin θ], [sin θ, cos θ]] in 2D; 3D rotations are more complex (Euler angles, quaternions)
• Scaling: S(sx, sy) = [[sx, 0], [0, sy]]; uniform scaling (sx = sy) preserves shape
• Reflection: Across line through origin; determinant = -1 (orientation-reversing)
• Shear: S(α) = [[1, α], [0, 1]]; skews shape parallel to x-axis
• Projection: Onto line or plane; projects 3D onto 2D for computer graphics

Homogeneous Coordinates and Affine Transformations

Linear transformations cannot represent translations. By using homogeneous coordinates (adding a fourth coordinate), we can represent translations, rotations, scaling, and perspective projections uniformly with 4×4 matrices. This is the foundation of computer graphics and robotics.

📚 How to Master Linear Algebra and Geometry

Recommended Approach

• Geometric Intuition First: Visualize vectors in ℝ² and ℝ³ before generalizing. Draw pictures of spans, linear combinations, and transformations.
• Master Matrix Multiplication: Understand it as composition of transformations, not just rows times columns. Practice with block matrices.
• Work Through Proofs: Linear algebra proofs are often constructive—they tell you how to build things (Gram-Schmidt, diagonalization). Follow each step carefully.
• Practice Computations: Compute eigenvalues, SVD, and determinants by hand for small matrices. The mechanics build intuition for the abstract theory.
• Connect to Applications: Learn PCA, SVD for image compression, and PageRank to see theory in action.

Recommended Resources

• Textbooks: Strang's Introduction to Linear Algebra, Axler's Linear Algebra Done Right, Hoffman & Kunze, Lay's Linear Algebra
• Video Lectures: Gilbert Strang's MIT 18.06, 3Blue1Brown's Essence of Linear Algebra series
• Interactive Tools: GeoGebra for visualizing transformations, Jupyter notebooks for computational experiments