202405181945
Status: #idea
Tags: Multiple Linear Regression

The Method Of Least Squares in Multiple Linear Regression

The logic is identical to the Simple Linear Regression case, except that we now need to represent everything in matrix form.

This changes the formulas since now we need to find our estimators using tensor calculus and stuff, but if you understand the logic in the single-variate case, doing it for multiple variables is just deriving the formulas using matrices all over again.

I will not bother going through the derivation mainly because I got a textbook that does it already, so I will just paste the formulas.

The Importance of $S{S}_{res}(\sum _{i=1}^n{e}^i)$

Pretty much all the formulas below are derived through some representation of the sum of error which we're trying to minimize. It is good to familiarize oneself with the ways to do so.

Formulas

$\hat{\beta }=$

$S{S}_{res}=S{S}_{Error}=\sum _{i=i}^n{e}_i^2=\sum _{i=i}^n({\mathit{Y}}_i-{\hat{Y}}_i{)}^2=$

From the equation for $\hat{\beta }$ and $S{S}_{res}$ we can simplify the equation for $S{S}_{res}$ to

The degree of freedoms of $S{S}_{res}$ in the multiple linear regression case is $n-k$ where $k$ is the number of parameters. This is nothing more than a generalization of the Simple Linear Regression which had $n-2$ degree of freedoms since we needed 2 error terms to be fixed to satisfy the normal equations.

Therefore we can compute the $M{S}_{res}$ as

The image above tells us that $S{S}_{res}$ is distributed according to a ${\chi }^2$ with $n-k$ degrees of freedom.

$S{S}_{total}=\sum _{i=1}^n(Y_i-\overline{Y}{)}^2=$

First $S{S}_T$ has $n-1$ degree of freedoms.

In matrix form this is written as:

Where $\overline{Y}$ is a scalar quantity.

Since we have a formula for $S{S}_{res}$ and a formula for $S{S}_T$ through the relation $S{S}_T=S{S}_{res}+S{S}_{reg}$ we can derive that:

$S{S}_{reg}=\sum _{i=1}^n({\hat{Y}}_i-\overline{Y}{)}^2=$

$S{S}_{reg}$ has how many degree of freedoms?

Recall that $D{F}_{total}=D{F}_{reg}+D{F}_{res}$, since we know that $D{F}_{total}$ is $n-1$ and that $D{F}_{res}$ is $n-k$ where $k$ is the number of parameters, it follows that $n-1-(n-k)$ or in a civilized world $k-1$ will be the number of degrees of freedom of $S{S}_{reg}$.

Therefore $M{S}_{reg}$ is:

Relevant Links

The Method Of Least Squares In Simple Linear Regression
Video on Multiple Linear Regression