Note! This notebook is a re-written version of the beginning of the Section 2/3 Day 8 notes from Prof. Higger. Feel free to consult those notes as supplementary material.

Line of Best Fit (1 dimension)

Sometimes, we have a set of points through which no straight line can pass. (There is no $b = ma$ where $m$ is the slope.)

That is, if we were to write the above data points of $\begin{bmatrix}1 \\ 1\end{bmatrix}, \begin{bmatrix}2 \\ 2\end{bmatrix}, \begin{bmatrix}3 \\ 4\end{bmatrix}$ in the form of:

$$ \begin{bmatrix}1 \\ 2 \\ 4\end{bmatrix} = m \begin{bmatrix}1 \\ 2 \\ 3\end{bmatrix} $$

There is no solution for this set of equations. (If $m$ is 1, then this will satisfy the first two but not the third.)

We can look at a graph of this:

So what we'll do instead is look for $\vec p$ that has the solution $\vec p = m \vec a$.

$\vec p$ has a few important qualities:

Recall (from the lecture on projections): Our equation for finding a projection $\vec p$ where $\vec b = \vec p + \vec e$ was:

$$ \vec p = \frac{b^Ta}{a^Ta}a $$

To find our line of best fit for the above example, instead of solving $b = ma$, we solve $p = ma$ where $\vec a = \begin{bmatrix}1 \\ 2 \\ 3\end{bmatrix}$ and $\vec b = \begin{bmatrix}1 \\ 2 \\ 4\end{bmatrix}$.

Plugging into the above equation:

$$ \vec p = \frac{1 * 1 + 2 * 2 + 4 * 3}{1 * 1 + 2 * 2 + 3 * 3} \begin{bmatrix}1 \\ 2 \\ 3\end{bmatrix} = \frac{17}{14}\begin{bmatrix}1 \\ 2 \\ 3\end{bmatrix} $$

So, we get:

$$ \frac{17}{14}\begin{bmatrix}1 \\ 2 \\ 3\end{bmatrix} = m \begin{bmatrix}1 \\ 2 \\ 3\end{bmatrix} $$

So $ m = \frac{17}{14}$.

Let's see what this looks like on the graph now!

ICA Question 1: Find the line of best fit for the below scatter points. (Using our equation above)

$$ \begin{bmatrix}-1 \\ 3\end{bmatrix}, \begin{bmatrix}1 \\ 4\end{bmatrix}, \begin{bmatrix}2 \\ 4\end{bmatrix} $$

Do this either by hand or using numpy.

Adding a non-zero intercept

Our equation so far only allows for lines that go through the origin. We'll make a very similar update to our equation that we made to the perceptron when we needed to allow for lines that don't go through the origin. Instead of looking for the equation $b = ma$, we'll look for the equation $b = m_0 + m_1a$ where $m_0$ will function as our intercept.

Just like we did with the perceptron, we'll use a "bias" term equivalent so that $m_0$ is "scaled" by 1 for every dimension.

For the problem above,

$$ \begin{bmatrix}3 \\ 4 \\ 4\end{bmatrix} = m \begin{bmatrix}-1 \\ 1 \\ 2\end{bmatrix} $$

becomes:

$$ \begin{bmatrix}3 \\ 4 \\ 4\end{bmatrix} = m_0\begin{bmatrix}1 \\ 1 \\ 1\end{bmatrix} + m_1\begin{bmatrix}-1 \\ 1 \\ 2\end{bmatrix} $$

Now, we can re-write this in terms of a matrix times a vector of slope factors: $\vec b = A\vec m$, giving us:

$$ \begin{bmatrix}3 \\ 4 \\ 4\end{bmatrix} = \begin{bmatrix}1 & -1 \\ 1 & 1 \\ 1 & 2\end{bmatrix} \begin{bmatrix}m_0 \\ m_1\end{bmatrix} $$

However, we know that $b = Am$ has no solutions! (There is no line that passes through all the points that we graphed), so we'll need to project $b$ into $A$, solving $\vec p = A \vec m$ instead.

(This means that we are looking for the linear combination of the columns of $A$ that gets the closest to $\vec b$.)

Luckily, we have an equation for this! (Again, from the projection lecture, lecture 8.)

$$ \vec p = A(A^TA)^{-1}A^Tb $$

We want to know what $\vec m$ was though! So that we can plot our line. To get this, we'll do the same manipulation that we did in 1 dimension:

$$ \begin{bmatrix} 3.07142857 \\ 3.78571429 \\ 4.14285714 \end{bmatrix} = m_0\begin{bmatrix}1 \\ 1 \\ 1\end{bmatrix} + m_1\begin{bmatrix}-1 \\ 1 \\ 2\end{bmatrix} $$

Now, we have a system of linear equations that we can solve for $m_0$ and $m_1$!

Polynomial of Best Fit

First, take a look at slide 11 from the lecture 8 slides from Prof. Higger:

image.png

Finally, please go explore the line_best_fit notebook that Prof. Higger posted for exploring Lines and polynomials of best fit!