types of matrices

1. Identity matrix: $I_n$

2. One matrix (matrix of ones): $J_n$

3. Diagonal matrix: $D$

4. Upper/lower triangular : $U, L$

5. Block matrices: $\begin{bmatrix} A & B \\C&D\end{bmatrix}$

6. invertible matrices are square matrices such that $AA^{-1} = A^{-1}A = I_n$ .

7. $(AB)^{-1} = B^{-1}A^{-1}$

8. an orthogonal matrix is a square matrix such that $A^{-1} = A^T$, and this is because the rows are orthonormal (think about it in terms of the dot product matrix product definition

9. a nilpotent matrix $N$ is constructed such that $N^k = 0$ for some positive, finite integer $k$

Upper triangular matrices

Formally, this is the definiition of upper triangular

• $Tv_i \in span(v_1, ...v_i)$

• or... $span(v_1, ...v_i)$ is invariant under $T$

think about this for a second and it should make sense

First, we can actually show that over the complex numbers, every operator has an upper triangular matrix. The proof is a little involved, so we won't show it here.

Upper triangular matrices are only invertible if the diagonals re all non-zero. this is because the determinant becomes zero otherwise.

Therefore, eigenvalues are just the entries on the diagonal for an upper-triangular matrix! This is because if you made any one of them zero, the matrix would become non-invertible.

Gram Matrix

We see a lot of $A^TA$ or $AA^T$, which is commonly known as gram matrix.

• Always PSD (super useful for convexity) [proof: $x^TA^TAx$ becomes norm]

• No guarantees on invertibility (it could have a low rank)

• Symmetric, which means that there exists eigenbasis, etc

• If $A$ is orthogonal, then $A^TA$ is an identity.

Interestingly, if $X$ represents a matrix of vector samples, $\frac{1}{n}X^TX$ is the empirical covariance matrix.

if $A$ has full column rank, then $A^TA$ is invertible. Why?

• Let $A^TAx = 0$

• From our four fundamental subspaces, we note that the null space of $A^T$ is the orthogonal complement of the range of $A$. Therefore, the range of $A$ can't be in the null space of $A^T$, so we know that $Ax = 0$

• Now, because $A$ is injective (full column rank), we conclude that $x = 0$. Therefore, we conclude that $A^TA$ is injective, and because $A^TA$ is an operator, we conclude that $A^TA$ is invertible.

This idea will be important when talking about left and right inverses in a second.

You can also conclude that $A^TA$ is symmetric, and this is a result of the dot product interpretation of matrix multiplication. This also means that $A^TA$ has an orthonormal basis of eigenvectors, which means that it can be expressed as $A^TA = QDQ^T$

Outer Product

If we have a nx1 vector $x$ (standard), we can compute the matrix $xx^T$, which is an nxn matrix. Each element is just $x_i * x_j$. This has rank $1$, because $xx^T$ is the SVD with a singular value.

In general, if we have an $n\times k$ matrix $x$, the $n\times n$ matrix of $xx^T$ has rank $k$ for the same reasons as outlined above.