Unit 3: Linear Algebra: Systems, Matrices, Eigenvalues, and Quadratic Forms

Solution of Linear Equations

Systems of linear equations are fundamental in mathematics and have wide applications. A system consists of one or more linear equations involving the same set of variables.

Homogeneous System of Linear Equations

A system of linear equations is called homogeneous if all the constant terms are zero. It can be written in matrix form as Ax = 0.

Definition: A homogeneous system of linear equations is a set of equations where each equation equals zero.

Properties:

• A homogeneous system always has at least one solution, known as the trivial solution, where all variables are zero (x₁ = 0, x₂ = 0, ..., x_n = 0).
• A homogeneous system can have non-trivial solutions (solutions where at least one variable is non-zero) if and only if the determinant of the coefficient matrix A is zero (det(A) = 0), or equivalently, if the rank of A is less than the number of variables.
• If det(A) ≠ 0, then the only solution is the trivial solution.

Non-Homogeneous System of Linear Equations

A system of linear equations is called non-homogeneous if at least one of the constant terms is non-zero. It can be written in matrix form as Ax = b, where b is a non-zero column vector.

Definition: A non-homogeneous system of linear equations is a set of equations where at least one equation has a non-zero constant term.

Consistency of a non-homogeneous system:

• A non-homogeneous system Ax = b is consistent (i.e., has at least one solution) if and only if the rank of the coefficient matrix A is equal to the rank of the augmented matrix [A|b]. (rank(A) = rank([A|b]))
• If rank(A) ≠ rank([A|b]), the system is inconsistent and has no solution.

Number of solutions for a consistent non-homogeneous system (Ax = b):

• If rank(A) = rank([A|b]) = n (number of variables), then the system has a unique solution.
• If rank(A) = rank([A|b]) < n (number of variables), then the system has infinitely many solutions. The number of free variables will be n - rank(A).

Partitioning of Matrices

Matrix partitioning involves dividing a matrix into smaller submatrices, called blocks or partitions. This can simplify operations or highlight certain structures within the matrix.

Definition and Examples

Definition: Matrix partitioning is the process of subdividing a matrix into smaller rectangular matrices called blocks or submatrices.

Example: Consider a 4×4 matrix A:

A = [[a11 a12 a13 a14],

     [a21 a22 a23 a24],

     [a31 a32 a33 a34],

     [a41 a42 a43 a44]]

This matrix can be partitioned into 2×2 blocks:

A = [[A11 A12],

     [A21 A22]]

Where:

A₁₁ = [[a₁₁ a₁₂], [a₂₁ a₂₂]]

A₁₂ = [[a₁₃ a₁₄], [a₂₃ a₂₄]]

A₂₁ = [[a₃₁ a₃₂], [a₄₁ a₄₂]]

A₂₂ = [[a₃₃ a₃₄], [a₄₃ a₄₄]]

Simple Properties of Partitioned Matrices

Operations with partitioned matrices generally follow the same rules as regular matrices, provided the blocks are conformable for the respective operations.

1. Addition of Partitioned Matrices:

• If A and B are two matrices partitioned identically, their sum C = A + B can be found by adding corresponding blocks:

C_ij = A_ij + B_ij

2. Multiplication of Partitioned Matrices:

• If A and B are conformably partitioned for multiplication, their product C = AB can be found by treating the blocks as elements and performing matrix multiplication. For example, if A is partitioned into m×p blocks and B into p×n blocks:

C_ij = ∑_k (A_ik B_kj)

• The inner dimensions of the blocks must be conformable for multiplication. For example, if A_ik is r×s and B_kj is s×t, then their product A_ik B_kj is r×t.

Characteristic Roots and Characteristic Vectors

Characteristic roots (eigenvalues) and characteristic vectors (eigenvectors) are fundamental concepts in linear algebra, revealing intrinsic properties of linear transformations and matrices.

Characteristic Roots (Eigenvalues)

Definition: A scalar λ (lambda) is called a characteristic root or eigenvalue of a square matrix A if there exists a non-zero vector x such that Ax = λx.

To find eigenvalues:

1. Form the characteristic equation: det(A - λI) = 0, where I is the identity matrix of the same dimension as A.
2. Solve the polynomial equation for λ. The roots of this equation are the eigenvalues.

Characteristic Vectors (Eigenvectors)

Definition: A non-zero vector x is called a characteristic vector or eigenvector corresponding to an eigenvalue λ of a square matrix A if Ax = λx.

To find eigenvectors:

1. For each eigenvalue λ found, substitute it back into the equation (A - λI)x = 0.
2. Solve the resulting system of linear equations for x. The non-trivial solutions are the eigenvectors corresponding to λ.

Properties of Characteristic Roots and Vectors

For a square matrix A of order n:

• The sum of the eigenvalues is equal to the trace of the matrix (sum of diagonal elements): ∑ λ_i = Tr(A).
• The product of the eigenvalues is equal to the determinant of the matrix: Π λ_i = det(A).
• If λ is an eigenvalue of A, then λ^k is an eigenvalue of A^k for any positive integer k.
• If λ is an eigenvalue of a non-singular matrix A, then 1/λ is an eigenvalue of A^-1.
• A square matrix A and its transpose A^T have the same eigenvalues.
• The eigenvalues of a symmetric matrix are always real.
• Eigenvectors corresponding to distinct eigenvalues are linearly independent.
• If a matrix A is triangular (upper or lower), its eigenvalues are its diagonal entries.

Applications of Characteristic Roots and Vectors

• Diagonalization of Matrices: Eigenvalues and eigenvectors are crucial for diagonalizing a matrix, which simplifies many matrix computations (e.g., A = PDP^-1, where D is a diagonal matrix of eigenvalues and P is a matrix whose columns are eigenvectors).
• Solving Systems of Differential Equations: Eigenvalues and eigenvectors provide solutions to linear systems of ordinary differential equations.
• Principal Component Analysis (PCA): Used in statistics and data analysis for dimensionality reduction.
• Stability Analysis: In dynamic systems, eigenvalues determine the stability of equilibria.

Cayley-Hamilton Theorem

Statement of the Theorem: Every square matrix satisfies its own characteristic equation.

If the characteristic equation of an n×n matrix A is given by p(λ) = c_nλⁿ + c_n-1λ^n-1 + ... + c₁λ + c₀ = 0, then according to the Cayley-Hamilton theorem, substituting A for λ gives:

c_nAⁿ + c_n-1A^n-1 + ... + c₁A + c₀I = 0

where I is the identity matrix of the same order as A, and 0 is the null matrix.

Applications of Cayley-Hamilton Theorem

1. Finding the Inverse of a Matrix:

• From the characteristic equation p(A) = 0, if c₀ ≠ 0 (i.e., det(A) ≠ 0), we can write:

c_nAⁿ + ... + c₁A = -c₀I

• Multiply by A^-1:

c_nA^n-1 + ... + c₁I = -c₀A^-1

A^-1 = (-1/c₀)(c_nA^n-1 + ... + c₁I)

2. Calculating Higher Powers of a Matrix:

• The theorem allows expressing higher powers of A (Aⁿ, Aⁿ⁺¹, etc.) as a linear combination of lower powers of A (A⁰, A¹, ..., A^n-1). This can simplify computations significantly.

Quadratic Forms

Quadratic forms are a specific type of mathematical expression that arises in various fields, including optimization, statistics, and geometry.

Definition of Quadratic Form

Definition: A quadratic form in n variables x₁, x₂, ..., x_n is a homogeneous polynomial of degree two of the form:

Q(x₁, x₂, ..., x_n) = ∑_i=1ⁿ ∑_j=1ⁿ a_ijx_ix_j

For example, a quadratic form in two variables x₁, x₂ is: Q(x₁, x₂) = a₁₁x₁² + 2a₁₂x₁x₂ + a₂₂x₂².

Matrix Representation of a Quadratic Form

Every quadratic form Q(x) can be uniquely represented in matrix form as x^TAx, where x is a column vector of variables and A is a symmetric matrix called the matrix of the quadratic form.

If x = [x₁, x₂, ..., x_n]^T, then A is an n×n symmetric matrix where:

• Diagonal elements: a_ii corresponds to the coefficient of x_i².
• Off-diagonal elements: a_ij = a_ji, and each equals half the coefficient of x_ix_j.

Example: For Q(x₁, x₂) = 3x₁² + 4x₁x₂ + 5x₂²,

x = [[x1],

     [x2]]

A = [[3  2],

     [2  5]]

Then Q(x) = x^TAx.

Canonical Form of a Quadratic Form

A quadratic form is said to be in canonical form if it contains only square terms, i.e., no cross-product terms (x_ix_j for i ≠ j). This can be achieved by an orthogonal transformation of variables, which is related to diagonalizing the matrix A of the quadratic form.

• If A is orthogonally diagonalizable (which it always is if A is symmetric), then there exists an orthogonal matrix P such that P^TAP = D, where D is a diagonal matrix of eigenvalues.
• By making the substitution x = Py, where y are new variables, the quadratic form x^TAx transforms to y^T(P^TAP)y = y^TDy.

y^TDy = λ₁y₁² + λ₂y₂² + ... + λ_ny_n²

This is the canonical form, where λ_i are the eigenvalues of A.

Nature of Quadratic Forms

The nature of a quadratic form (positive definite, negative definite, etc.) is determined by the signs of the eigenvalues of its associated symmetric matrix A.