| Optimization basics | Basics |
| Quadratic Forms & PSD | Basics |
| Hessian Matrix | Basics |
| Jacobian Matrix | Basics |
| Matrix Calculus: fundamentals | Backprop |
| Matrix Derivative Practicals | Backprop |
| Matrix Calculus: Common Forms | Backprop |
| Gradient Techniques (algorithms) | Backprop |
| Natural Gradient | Second Order |
| Newton’s Method | Second Order |
| Conjugate Gradient Descent | Second Order |
| Hessian-Free Gradients | Second Order |
| Dual Gradient Descent | Constrained |
| Constrained Optimization: an intro | Constrained |
| Constrained Optimization | Constrained |
| CEM and CMA | Gradient Free |
| Overdetermined Linear Equations and Least squares (left inverses) | Closed Form |
| Gauss-Newton Method + Regularized Least Squares | Closed Form |
| Underdetermined Linear Equations (right inverses) | Closed Form |