Common patterns

• If you see something like $a_{ij} = x_ix_j$, think immediately of $xx^T$

• When you see something like $x^Ty$, immediately think "scalar", and therefore $x^Ty = y^Tx$

• when taking the derivative WRT parameter $\theta$, make sure that it's out front! So something like $x^T\theta$ should be swapped to $\theta^T x$

• If $A$ is symmetric, then $(Av)^T = v^TA$. Kinda obvious but it can show up!

• If $A$ is symmetric, then $(A^{-1})^T = A^{-1}$

• every vector in $\mathbb{R}^n$ is a column vector by default

Exploiting the heck out of distributive

Distributive property will save you a lot of headache. It works on any linear operation, which means that matrices are fair game. Transposes are distributive as well, and dot products are definitely distributive, both left and right hand side.

Therefore, something like this is possible:

And something like this is really helpful:

Comparisons of vectors, matrices

• Vector inequality doesn’t make much sense: you need to norm it, or otherwise compute a scalar property. Then, you can use properties like Cauchy Schwartz, etc.

• Matrix inequality also doesn’t make much sense, but you can compute a scalar property using a quadratic form, and sometimes you get nice inequalities.

• If you say $A \geq B$, usually you mean element-wise

• If you say $A\succeq B$, usually it means that $A - B$ is PSD and $B - A$ is ND. It’s a tricky distinction!

Change of coordinates

This is actually pretty obvious but also a little tricky. If we wanted to map from the canonical basis $e_1,…, e_n$ to another basis $t_1, …, t_n$, then we can make a matrix $T$ whose columns is the basis. Then, $Tx$ would be expressing the other basis in terms of the canonical basis. Therefore, $T^{-1}x$ is expressing the canonical basis in terms of the other basis. It’s flipped from intuition, so be careful!

Therefore, if you have a canonical function $A$ and another basis that your $\tilde x, \tilde y$ are in, then you can define

You can imagine going through a “portal” to the canonical place, applying the function, and the going through an inverse portal to the other basis.

Matrix to summations, and vice versa

• when you think $z^T A z$, think $\sum_{i, j} A_{ij}z_i z_j$. This is easily derived. And you can actually use this to derive the gradient of the quadratic form
  • conversely, if you ever see that summation, immediately think: $z^TAz$

• Another trick is that if you have something like $(x^Tz)^2$, you can write it out as the product of two sums, and because the summations use different indices, you can merge them into a single double sum.
  • again, conversely, if you ever see $\sum_{i, j} x_ix_jz_iz_j$, think immediately $(x^Tz)^2$.

Completing the square

Recall that the complete the square in scalar form is

We can also rewrite this as matrix multiplication too!