Note! This notebook is a re-written version of the beginning of the Section 2/3 Day 8 notes from Prof. Higger. Feel free to consult those notes as supplementary material.
# so that I can embed some nice graphs for you
import matplotlib.pyplot as plt
# for some linear algebra magic
import numpy as np
Sometimes, we have a set of points through which no straight line can pass. (There is no $b = ma$ where $m$ is the slope.)
That is, if we were to write the above data points of $\begin{bmatrix}1 \\ 1\end{bmatrix}, \begin{bmatrix}2 \\ 2\end{bmatrix}, \begin{bmatrix}3 \\ 4\end{bmatrix}$ in the form of:
$$ \begin{bmatrix}1 \\ 2 \\ 4\end{bmatrix} = m \begin{bmatrix}1 \\ 2 \\ 3\end{bmatrix} $$There is no solution for this set of equations. (If $m$ is 1, then this will satisfy the first two but not the third.)
We can look at a graph of this:
plt.plot([1, 2, 3], [1, 2, 4], "o")
# try to put in a line that fits them
plt.plot([0, 1, 2, 3, 4], [0, 1, 2, 3, 4], label="line of not best fit")
plt.grid()
plt.ylabel("b")
plt.xlabel("a")
plt.legend()
plt.show()
So what we'll do instead is look for $\vec p$ that has the solution $\vec p = m \vec a$.
$\vec p$ has a few important qualities:
Recall (from the lecture on projections): Our equation for finding a projection $\vec p$ where $\vec b = \vec p + \vec e$ was:
$$ \vec p = \frac{b^Ta}{a^Ta}a $$To find our line of best fit for the above example, instead of solving $b = ma$, we solve $p = ma$ where $\vec a = \begin{bmatrix}1 \\ 2 \\ 3\end{bmatrix}$ and $\vec b = \begin{bmatrix}1 \\ 2 \\ 4\end{bmatrix}$.
Plugging into the above equation:
$$ \vec p = \frac{1 * 1 + 2 * 2 + 4 * 3}{1 * 1 + 2 * 2 + 3 * 3} \begin{bmatrix}1 \\ 2 \\ 3\end{bmatrix} = \frac{17}{14}\begin{bmatrix}1 \\ 2 \\ 3\end{bmatrix} $$So, we get:
$$ \frac{17}{14}\begin{bmatrix}1 \\ 2 \\ 3\end{bmatrix} = m \begin{bmatrix}1 \\ 2 \\ 3\end{bmatrix} $$So $ m = \frac{17}{14}$.
Let's see what this looks like on the graph now!
def line_func(min_num, max_num, slope):
xs = list(range(min_num, max_num))
ys = [x * slope for x in xs]
return (xs, ys)
plt.plot([1, 2, 3], [1, 2, 4], "o")
# try to put in a line that fits them
xs, ys = line_func(0, 5, 17/14)
plt.plot(xs, ys, label="line of best fit")
plt.grid()
plt.ylabel("b")
plt.xlabel("a")
plt.legend()
plt.show()
ICA Question 1: Find the line of best fit for the below scatter points. (Using our equation above)
$$ \begin{bmatrix}-1 \\ 3\end{bmatrix}, \begin{bmatrix}1 \\ 4\end{bmatrix}, \begin{bmatrix}2 \\ 4\end{bmatrix} $$Do this either by hand or using numpy.
plt.plot([-1, 1, 2], [3, 4, 4], "o")
plt.grid()
plt.ylabel("b")
plt.xlabel("a")
plt.axhline(y = 0)
plt.axvline(x = 0)
plt.xlim(-4, 4)
plt.ylim(-4, 5)
#plt.legend()
plt.show()
Our equation so far only allows for lines that go through the origin. We'll make a very similar update to our equation that we made to the perceptron when we needed to allow for lines that don't go through the origin. Instead of looking for the equation $b = ma$, we'll look for the equation $b = m_0 + m_1a$ where $m_0$ will function as our intercept.
Just like we did with the perceptron, we'll use a "bias" term equivalent so that $m_0$ is "scaled" by 1 for every dimension.
For the problem above,
$$ \begin{bmatrix}3 \\ 4 \\ 4\end{bmatrix} = m \begin{bmatrix}-1 \\ 1 \\ 2\end{bmatrix} $$becomes:
$$ \begin{bmatrix}3 \\ 4 \\ 4\end{bmatrix} = m_0\begin{bmatrix}1 \\ 1 \\ 1\end{bmatrix} + m_1\begin{bmatrix}-1 \\ 1 \\ 2\end{bmatrix} $$Now, we can re-write this in terms of a matrix times a vector of slope factors: $\vec b = A\vec m$, giving us:
$$ \begin{bmatrix}3 \\ 4 \\ 4\end{bmatrix} = \begin{bmatrix}1 & -1 \\ 1 & 1 \\ 1 & 2\end{bmatrix} \begin{bmatrix}m_0 \\ m_1\end{bmatrix} $$However, we know that $b = Am$ has no solutions! (There is no line that passes through all the points that we graphed), so we'll need to project $b$ into $A$, solving $\vec p = A \vec m$ instead.
(This means that we are looking for the linear combination of the columns of $A$ that gets the closest to $\vec b$.)
Luckily, we have an equation for this! (Again, from the projection lecture, lecture 8.)
$$ \vec p = A(A^TA)^{-1}A^Tb $$# now, let's let numpy help us
A = np.array([[1, -1], [1, 1], [1, 2]])
b = np.array([[3, 4, 4]]).T
AtransAinv = np.linalg.inv(A.T @ A)
print(AtransAinv)
p = A @ AtransAinv @ A.T @ b
print("Answer: \n", p)
[[ 0.42857143 -0.14285714] [-0.14285714 0.21428571]] Answer: [[3.07142857] [3.78571429] [4.14285714]]
We want to know what $\vec m$ was though! So that we can plot our line. To get this, we'll do the same manipulation that we did in 1 dimension:
$$ \begin{bmatrix} 3.07142857 \\ 3.78571429 \\ 4.14285714 \end{bmatrix} = m_0\begin{bmatrix}1 \\ 1 \\ 1\end{bmatrix} + m_1\begin{bmatrix}-1 \\ 1 \\ 2\end{bmatrix} $$Now, we have a system of linear equations that we can solve for $m_0$ and $m_1$!
# you can even do this using sympy and two of our equations
import sympy
x = np.array([[1, -1, 3.07142857], [1, 1, 3.78571429]])
sympy.Matrix(x).rref()[0]
def line_func2(min_num, max_num, m1, m0):
xs = list(range(min_num, max_num))
ys = [x * m1 + m0 for x in xs]
return (xs, ys)
plt.plot([-1, 1, 2], [3, 4, 4], "o")
# try to put in a line that fits them
xs, ys = line_func2(-2, 4, 0.35714286, 3.42857143)
plt.plot(xs, ys, label="line of best fit")
plt.grid()
plt.ylabel("b")
plt.xlabel("a")
plt.legend()
plt.show()
First, take a look at slide 11 from the lecture 8 slides from Prof. Higger:
Finally, please go explore the line_best_fit notebook that Prof. Higger posted for exploring Lines and polynomials of best fit!