202405170137
Status: #idea
Tags: Machine Learning

Supervised Learning

Fundamentally the idea of supervised learning is simply to fit a curve to data.

That is literally it.
Everything else is just mathematical jargon and calculus (to find the right parameters and what not.) I lied, there's also some Probability and Statistics in there as well as Linear Algebra...

The rest is just math to derive your parameters.

Supervised learning can be summarized as the following equations:

$y$ here is a vector of observations, $f$ is whatever class of mathematical functions we think is a appropriate to fit in our model. In the case of Simple Linear Regression and Multiple Linear Regression they are linear functions, but in the case of Deep Learning it will invariably be some kind of neural network architecture. Finally we have $\varphi$ which represents all our parameters.

When someone says with the chest puffed that they are "training" a model or that the model "learned" from the data, it is nothing more than fancy speak for "we mathe-magically tweaked the vector of parameters $\varphi$ until it gave us results that made sense."

The above sentence might change in the future with the advent of things like KANN which allow us to train the activation functions as well, but that is besides the point for now.

Methodology

Well, it depends, and if you are hardy enough for some of the more well-known models you could actually sit down to derive the best parameters (those that minimize the loss) by hand.

But in Deep Learning the models we are dealing with are way too big for that, and even if they weren't we have computers to do the job. So the idea is more or less always something along the line of:

1. Initialize the parameters at random values (Making sure those values are in the right ballpark can make the difference between a model that is impossible to train, and one that trains in seconds)
2. Compute the loss function $L$ for those randomly initialized parameters (ie: Least Square Error)
3. Compute the gradient for that loss function (it represents the direction of steepest ascent)
4. Back-propagate it to your parameters (now they all should have data that shows how to increase the loss)
5. Take a step proportional to your Learning Rate in the direction opposite to the gradient.
6. Repeat until either:
   • Gradient is $0$ and no more improvement can be made
   • We've completed to train for all the allocated epochs
   • We have run out of time
   • We have run out of resources
   • The model's loss has been stagnating for a given number of epochs
   • The model's loss has actually started going up
   • etc.

This is all there is to it fundamentally, we just want to find the parameters often called weights, which minimize our loss function or in math-speak:

Since we never know the actual population statistics so-to speak of our model, we need to approximate them. And the above equation just means "the vector of parameters that best estimate $\varphi$ is given by the set of parameters that minimize the loss function."

The rest is just the math to derive the parameters, and whatnot.

Finding architectures is a rough task as well.

Generative vs Discriminative Models

Discriminative Models--what we presented above--are models which go the intuitive way of using the regressors or independent variables to predict the dependent variables. Generative Models on the other hand, choose the unorthodox approach of "generating" the measurements/regressors that could have lead to the observation from the observations; essentially inverting the direction.

So while discriminative models take the form $$
\boldsymbol y = f[\boldsymbol x, \phi]

\boldsymbol x = g[\boldsymbol y,\phi]

y = g^{-1}[\boldsymbol x, \phi]