\documentclass[11pt]{article}
%\usepackage{geometry}                % See geometry.pdf to learn the layout options. There are lots.
%\geometry{letterpaper}                   % ... or a4paper or a5paper or ... 
%\geometry{landscape}                % Activate for for rotated page geometry
%\usepackage[parfill]{parskip}    % Activate to begin paragraphs with an empty line rather than an indent
\usepackage{graphicx}
\usepackage{amssymb}
\usepackage{epstopdf}
\usepackage{amsmath}
\usepackage{amsthm}
\usepackage{hyperref}
\usepackage{color}
\DeclareGraphicsRule{.tif}{png}{.png}{`convert #1 `dirname #1`/`basename #1 .tif`.png}

% margin notes
\usepackage{marginnote}
\usepackage[top=2cm, bottom=2cm, outer=7cm, inner=2cm, heightrounded, marginparwidth=5cm, marginparsep=1cm]{geometry}
%\geometry{letterpaper}                   % ... or a4paper or a5paper or ... 
\renewcommand*{\marginfont}{\footnotesize}

\input{../../misc/macros.tex}



\title{Lecture 1, Part 1: Linear regression}
\author{Roger Grosse}
\date{}

\begin{document}
\maketitle

\section{Introduction}

Let's jump right in and look at our first machine learning algorithm, {\bf linear regression}. In regression, we are interested in predicting a scalar-valued target, such as the price of a stock. By linear, we mean that the target must be predicted as a linear function of the inputs. This is a kind of supervised learning algorithm; recall that, in supervised learning, we have a collection of training examples labeled with the correct outputs.

Regression is an important problem in its own right. But today's discussion will also highlight a number of themes which will recur throughout the course:
\begin{itemize}
\item Formulating a machine learning task mathematically as an optimization problem.
\item Thinking about the data points and the model parameters as vectors.
\item Solving the optimization problem using two different strategies: deriving a closed-form solution, and applying gradient descent. These two strategies are how we will derive nearly all of the learning algorithms in this course.
\item Writing the algorithm in terms of linear algebra, so that we can think about it more easily and implement it efficiently in a high-level programming language.
\item Making a linear algorithm more powerful using basis functions, or features.
\item Analyzing the generalization performance of an algorithm, and in particular the problems of overfitting and underfitting.
\end{itemize}


\subsection{Learning goals}

\begin{itemize}
\item Know what objective function is used in linear regression, and how it is motivated.
%\item Know how we can get rid of the bias term using a ``dummy'' feature.
\item Derive both the closed-form solution and the gradient descent updates for linear regression.
\item Write both solutions in terms of matrix and vector operations.
\item Be able to implement both solution methods in Python.
\item Know how linear regression can learn nonlinear functions using feature maps.
\item What is meant by generalization, overfitting, and underfitting?  How can we measure generalization performance in practice?
\end{itemize}

\iffalse
\subsection{Resources}

\begin{itemize}
\item This lecture doesn't line up well with the Coursera course. However, there are tons of good introductions to linear regression available for free online. For a listing, see the Metacademy concepts ``linear regression'' and ``linear regression: closed-form solution.''
\item Suggested background:
\begin{itemize}
\item The very basics of linear algebra: Metacademy concepts ``matrix multiplication'', ``matrix transpose'', and ``matrix inverse''
\item Partial derivatives, since we'll use these to derive both of the solution methods. See Metacademy concept ``partial derivatives''.
\end{itemize}
\end{itemize}
\fi


\section{Problem setup}

In order to formulate a learning problem mathematically, we need to define two things: a model and a loss function. The {\bf model}, or {\bf architecture} defines the set of allowable {\bf hypotheses}, or functions that compute predictions from the inputs. In the case of linear regression, the model simply consists of linear functions. Recall that a linear function of $\nfeat$ inputs is parameterized in terms of $\nfeat$ coefficients, which we'll call the {\bf weights}, and an intercept term, which we'll call the {\bf bias}. Mathematically, this is written as:
\begin{equation}
\prediction = \sum_\featIdx \weightJ{\featIdx} \inputJ{\featIdx} + \bias. \label{eqn:model}
\end{equation}
Figure \ref{fig:data-space-weight-space} shows two ways to visualize linear models. In this case, the data are one-dimensional, so the model reduces to simply $\prediction = \weightUni \inputUni + \bias$. On one side, we have the {\bf data space}, or {\bf input space}, where $\target$ is plotted as a function of $\inputUni$. Three different possible linear fits are shown. On the other side, we have {\bf weight space}, where the corresponding pairs $(\weightUni, \bias)$ are plotted.\marginnote{You should study these figures and try to understand how the lines in the left figure map onto the X's on the right figure. Think back to middle school. Hint: $\weightUni$ is the slope of the line, and $\bias$ is the y-intercept.}

\begin{figure}
\begin{center}
\includegraphics[width=0.6 \textwidth]{figures/data_space_weight_space.png}
\end{center}
\caption{Three possible hypotheses for a linear regression model, shown in data space and weight space.}
\label{fig:data-space-weight-space}
\end{figure}

Clearly, some of these linear fits are better than others. In order to quantify how good the fit is, we define a {\bf loss function}. This is a function $\loss(\prediction, \target)$ which says how far off the prediction $\prediction$ is from the target $\target$. In linear regression, we use {\bf squared error}, defined as
\begin{equation}
\loss(\prediction, \target) = \frac{1}{2} (\prediction - \target)^2. \label{eqn:loss}
\end{equation}
This is small when $\prediction$ and $\target$ are close together, and large when they are far apart. \marginnote{Why is there the factor of $1/2$ in front?  It just makes the calculations convenient.} In general, the value $\prediction - \target$ is known as the {\bf residual}, and we'd like the residuals to be close to zero. 

When we combine our model and loss function, we get an optimization problem, where we are trying to minimize a {\bf cost function} with respect to the model parameters (i.e.~the weights and bias). The cost function is simply the loss, averaged over all the training examples. When we plug in the model definition (Eqn.~\ref{eqn:model}), we get the following cost function:
\begin{align}
\cost(\weightJ{1}, \ldots, \weightJ{\nfeat}, \bias) &= \frac{1}{\ndata} \sum_{\dataIdx=1}^\ndata \loss(\predictionI{\dataIdx}, \targetI{\dataIdx}) \label{eqn:cost-from-loss} \\
&= \frac{1}{2 \ndata} \sum_{\dataIdx=1}^\ndata \left( \predictionI{\dataIdx} - \targetI{\dataIdx} \right)^2 \\
&= \frac{1}{2 \ndata} \sum_{\dataIdx=1}^\ndata \left( \sum_\featIdx \weightJ{\featIdx} \inputIJ{\dataIdx}{\featIdx} + \bias - \targetI{\dataIdx} \right)^2 \label{eqn:cost}
\end{align}
Our goal is to choose $\weightJ{1}, \ldots, \weightJ{\nfeat}$ and $\bias$ to minimize $\cost$. Note the difference between the loss function and the cost function. The loss is a function of the predictions and targets, while the cost is a function of the model parameters.\marginnote{The distinction between loss functions and cost functions will become clearer in a later lecture, when the cost function is augmented to include more than just the loss --- it will also include a term called a regularizer which encourages simpler hypotheses.} The cost function is visualized in Figure \ref{fig:cost-function}.

\begin{figure}
%\begin{center}
\makebox[0.8 \paperwidth][l]
{
\includegraphics[width=0.25 \paperwidth]{figures/residuals.png}
\includegraphics[width=0.25 \paperwidth]{figures/contours.png}
\hspace{-2em}
\includegraphics[width=0.3 \paperwidth]{figures/surface.png}
}
%\end{center}
\caption{{\bf Left:} three hypotheses for a regression dataset. {\bf Middle:} Contour plot of least-squares cost function for the regression problem. Colors of the points match the hypotheses. {\bf Right:} Surface plot matching the contour plot. Surface plots are usually hard to interpret, so we won't look at them very often.}
\label{fig:cost-function}
\end{figure}


\section{Solving the optimization problem}

In order to solve the optimization problem, we'll need the concept of partial derivatives. If you haven't seen these before, then you should go learn about them, on Khan Academy.\footnote{\url{https://www.khanacademy.org/math/calculus-home/multivariable-calculus/multivariable-derivatives\#partial-derivatives}} Just as a quick recap, suppose $f$ is a function of  $x_1, \ldots, x_D$. Then the partial derivative $\partial f / \partial x_i$ says in what way the value of f changes if you increase $x_i$ by a small amount, while holding the rest of the arguments fixed. We can evaluate partial derivatives using the tools of single-variable calculus: to compute $\partial f / \partial x_i$ simply compute the (single-variable) derivative with respect to $x_i$, treating the rest of the arguments as constants. 

Whenever we want to solve an optimization problem, a good place to start is to compute the partial derivatives of the cost function. Let's do that in the case of linear regression. Applying the chain rule for derivatives to Eqn.~\ref{eqn:cost}, we get
\begin{align}
\frac{\partial \cost}{\partial \weightJ{\featIdx}} &= \frac{1}{\ndata} \sum_{\dataIdx=1}^\ndata \inputIJ{\dataIdx}{\featIdx} \left( \sum_{\featIdxTwo} \weightJ{\featIdxTwo} \inputIJ{\dataIdx}{\featIdxTwo} + \bias - \targetI{\dataIdx} \right)  \label{eqn:weight-derivatives-full} \\
\frac{\partial \cost}{\partial \bias} &= \frac{1}{\ndata} \sum_{\dataIdx=1}^\ndata \left( \sum_{\featIdxTwo} \weightJ{\featIdxTwo} \inputIJ{\dataIdx}{\featIdxTwo} + \bias - \targetI{\dataIdx} \right). \label{eqn:bias-derivative-full}
\end{align}
It's possible to simplify this a bit --- notice that part of the term in parentheses is simply the prediction.\marginnote{It's always a good idea to try to simplify equations by finding familiar terms.} The partial derivatives can be rewritten as:
\begin{align}
\frac{\partial \cost}{\partial \weightJ{\featIdx}} &= \frac{1}{\ndata} \sum_{\dataIdx=1}^\ndata \inputIJ{\dataIdx}{\featIdx} ( \predictionI{\dataIdx} - \targetI{\dataIdx} ) \label{eqn:weight-derivative} \\
\frac{\partial \cost}{\partial \bias} &= \frac{1}{\ndata} \sum_{\dataIdx=1}^\ndata \predictionI{\dataIdx} - \targetI{\dataIdx}. \label{eqn:bias-derivative}
\end{align}
Now, it's good practice to do a sanity check of the derivatives. For instance, suppose we overestimated all of the targets. Then we should be able to improve the predictions by decreasing the bias, while holding all of the weights fixed. Does this work out mathematically?  Well, the residuals $\predictionI{\dataIdx} - \targetI{\dataIdx}$ will be positive, so based on Eqn.~\ref{eqn:bias-derivative},  $\partial \cost / \partial \bias$ will be positive. This means increasing the bias will increase $\cost$, and deceasing the bias will decrease $\cost$ --- which matches up with our expectation. So Eqn.~\ref{eqn:bias-derivative} is plausible. Try to come up with a similar sanity check for $\partial \cost / \partial \weightJ{\featIdx}$.\marginnote{Later in this course, we'll introduce a more powerful way to test partial derivative computations, but you should still get used to doing sanity checks on all your computations!}

Now how do we use these partial derivatives?  Let's discuss the two methods which we will use throughout the course.

\subsection{Direct solution}
\label{sec:partial-derivatives-zero}

One way to compute the minimum of a function is to set the partial derivatives to zero. Recall from single variable calculus that (assuming a function is differentiable) the minimum $x^\star$ of a function $f$ has the property that the derivative $\deriv f / \deriv x$ is zero at $x = x^\star$. Note that the converse is not true: if $\deriv f / \deriv x = 0$, then $x^\star$ might be a maximum or an inflection point, rather than a minimum. But the minimum can only occur at points that have derivative zero. 

An analogous result holds in the multivariate case: if $f$ is differentiable, then all of the partial derivatives $\partial f / \partial x_i$ are zero at the minimum. The intuition is simple: if $\partial f / \partial x_i$ is positive, then one can decrease $f$ slightly by decreasing $x_i$ slightly. Conversely, if $\partial f / \partial x_i$ is negative, then one can decrease $f$ slightly by increasing $x_i$ slightly. In either case, this implies we're not at the minimum. Therefore, if the minimum exists (i.e.~$f$ doesn't keep growing as $x$ goes to infinity), it occurs at a {\bf critical point}, i.e.~a point where the partial derivatives are zero. This gives us a strategy for finding minima: set the partial derivatives to zero, and solve for the parameters. This method is known as {\bf direct solution}.

Let's apply this to linear regression. For simplicity, let's assume the model doesn't have a bias term. (We actually don't lose anything by getting rid of the bias. Just add a ``dummy'' input $\inputJ{0}$ which always takes the value 1; then the weight $\weightJ{0}$ acts as a bias.) We simplify Eqn.~\ref{eqn:weight-derivatives-full} to remove the bias, and set the partial derivatives to zero:
\begin{equation}
\frac{\partial \cost}{\partial \weightJ{\featIdx}} = \frac{1}{\ndata} \sum_{\dataIdx=1}^\ndata \inputIJ{\dataIdx}{\featIdx} \left( \sum_{\featIdxTwo=1}^\nfeat \weightJ{\featIdxTwo} \inputIJ{\dataIdx}{\featIdxTwo} - \targetI{\dataIdx} \right) = 0
\end{equation}
Since we're trying to solve for the weights, let's pull these out:
\begin{equation}
\frac{\partial \cost}{\partial \weightJ{\featIdx}} = \frac{1}{\ndata} \sum_{\featIdxTwo=1}^\nfeat \left( \sum_{\dataIdx=1}^\ndata \inputIJ{\dataIdx}{\featIdx} \inputIJ{\dataIdx}{\featIdxTwo} \right) \weightJ{\featIdxTwo} -  \frac{1}{\ndata} \sum_{\dataIdx=1}^\ndata \inputIJ{\dataIdx}{\featIdx} \targetI{\dataIdx} = 0 \\
\end{equation}
The details of this equation aren't important; what's important is that we've wound up with a system of $\nfeat$ linear equations in $\nfeat$ variables. In other words, we have the system of linear equations
\begin{equation}
\sum_{\featIdxTwo=1}^\nfeat A_{\featIdx \featIdxTwo} \weightJ{\featIdxTwo} - c_{\featIdx} = 0 \qquad \forall \featIdx \in \{1, \ldots, \nfeat \},
\end{equation}
where $A_{\featIdx \featIdxTwo} = \frac{1}{\ndata} \sum_{\dataIdx=1}^\ndata \inputIJ{\dataIdx}{\featIdx} \inputIJ{\dataIdx}{\featIdxTwo}$ and $c_{\featIdx} = \frac{1}{\ndata} \sum_{\dataIdx=1}^\ndata \inputIJ{\dataIdx}{\featIdx} \targetI{\dataIdx}$. As computer scientists, we're done, because this gives us an algorithm for finding the optimal regression weights: we first compute all the values $A_{\featIdx \featIdxTwo}$ and $c_{\featIdx}$, and then solve the system of linear equations using a linear algebra library such as NumPy. (We'll give an implementation of this later in this lecture.)

Note that the solution we just derived is \emph{very} particular to linear regression. In general, the system of equations will be \emph{nonlinear}, and except in rare cases, systems of nonlinear equations don't have closed form solutions. Linear regression is very unusual, in that it has a closed-form solution. We'll only be able to come up with closed form solutions for a handful of the algorithms we cover in this course.


\subsection{Gradient descent}

Now let's minimize the cost function a different way: {\bf gradient descent}. This is an example of an {\bf iterative algorithm}, which means that we apply a certain update rule over and over again, and if we're lucky, our {\bf iterates} will gradually improve according to our objective function. To do gradient descent, we initialize the weights to some value (e.g.~all zeros), and repeatedly adjust them in the direction that most decreases the cost function. If we visualize the cost function as a surface, so that lower is better, this is the direction of {\bf steepest descent}. We repeat this procedure until the iterates {\bf converge}, or stop changing much. (Or, in practice, we often run it until we get tired of waiting.) If we're lucky, the final iterate will be close to the optimum.

In order to make this mathematically precise, we must introduce the {\bf gradient}, the direction of steepest ascent (i.e.~fastest increase) of a function. The entries of the gradient vector are simply the partial derivatives with respect to each of the variables:
\begin{equation}
\frac{\partial \cost}{\partial \weights} = \begin{pmatrix} \frac{\partial \cost}{\partial \weightJ{1}} \\ \vdots \\ \frac{\partial \cost}{\partial \weightJ{\nfeat}} \end{pmatrix}
\end{equation}
The reason that this formula gives the direction of steepest ascent is beyond the scope of this course. (You would learn about it in a multivariable calculus class.) But this suggests that to decrease a function as quickly as possible, we should update the parameters in the direction opposite the gradient. 

We can formalize this using the following update rule, which is known as {\bf gradient descent}:
\begin{equation}
\weights \gets \weights - \lrate \frac{\partial \cost}{\partial \weights}, \label{eqn:gd-formula-vec}
\end{equation}
or in terms of coordinates,
\begin{equation}
\weightJ{\featIdx} \gets \weightJ{\featIdx} - \lrate \frac{\partial \cost}{\partial \weightJ{\featIdx}}.
\end{equation}
The symbol $\gets$ means that the left-hand side is updated to take the value on the right-hand side; the constant $\lrate$ is known as a {\bf learning rate}. The larger it is, the larger a step we take. We'll talk in much more detail later about how to choose a learning rate, but in general it's good to choose a small value such as 0.01 or 0.001. If we plug in the formula for the partial derivatives of the regression model (Eqn.~\ref{eqn:weight-derivative}), we get the update rule:
\marginnote{In practice, we rarely if ever go through this last step. From a software engineering perspective, it's better to write our code in a modular way, where one function computes the gradient, and another function implements gradient descent, taking the gradient as given.}
\begin{equation}
\weightJ{\featIdx} \gets \weightJ{\featIdx} - \lrate  \frac{1}{\ndata} \sum_{\dataIdx=1}^\ndata \inputJ{\featIdx} ( \predictionI{\dataIdx} - \targetI{\dataIdx} ) \label{eqn:expanded-gd-update}
\end{equation}

So we just repeat this update lots of times. What does gradient descent give us in the end?  For analyzing iterative algorithms, it's useful to look for {\bf fixed points}, i.e.~points where the iterate doesn't change. By inspecting Eqn.~\ref{eqn:gd-formula-vec}, setting the left-hand side equal to the right-hand side, we see that the fixed points occur where $\partial \cost / \partial \weights = 0$. Since we know the gradient must be zero at the optimum, this is an encouraging sign that maybe it will converge to the optimum. But there are lots of things that could go wrong, such as divergence or local optima; we'll look at these in more detail in a later lecture.\marginnote{Lecture 9 discusses optimization issues.}



%Figure \TODO{} shows an example of optimizing the linear regression objective using gradient descent. The weights gradually approach the optimal value.

You might ask: by setting the partial derivatives to zero, we compute the exact solution. With gradient descent, we never actually reach the optimum, but merely approach it gradually. Why, then, would we ever prefer gradient descent?  Two reasons:
\begin{enumerate}
\item We can only solve the system of equations explicitly for a handful of models. By contrast, we can apply gradient descent to any model for which we can compute the gradient. This is usually pretty easy to do efficiently. Importantly, it can usually be done \emph{automatically}, so software packages like Theano and TensorFlow can save us from ever having to compute partial derivatives by hand.
\item Solving a large system of linear equations can be expensive, possibly many orders of magnitude more expensive than a single gradient descent update. Therefore, gradient descent can sometimes find a reasonable solution much faster than solving the linear system. Therefore, gradient descent is often more practical than computing exact solutions, even for models where we are able to derive the latter.
\end{enumerate}
For these reasons, gradient descent will be our workhorse throughout the course. We will use it to train almost all of our models, with the exception of a handful for which we can derive exact solutions.

\section{Vectorization}

Now it's time to bring in linear algebra. We're going to rewrite the linear regression model, as well as both solution methods, in terms of operations on matrices and vectors. This process is known as {\bf vectorization}. There are two reasons for doing this:
\marginnote{Vectorization takes a lot of practice to get used to. We'll cover a lot of examples in the first few weeks of the course. I'd recommend practicing these until they start to feel natural.}
\begin{enumerate}
\item The formulas can be much simpler, more compact, and more readable in this form.
\item Vectorized code can be much faster than explicit \verb+for+-loops, for several reasons.
\begin{itemize}
\item High-level languages like Python can introduce a lot of interpreter overhead, and if we explicitly write a for-loop corresponding to Eqn.~\ref{eqn:expanded-gd-update}, this might be 10-100 times slower than the C equivalent. If we instead write the algorithm in terms of a much smaller number of linear algebra operations, then it can perform the same computations much faster with minimal interpreter overhead.
\item Since linear algebra is used all over the place, linear algebra libraries have been extremely well optimized for various computer architectures. Hence, they use much more efficient memory-access patterns than a \naive for-loop, even one written in C.
\item Matrix multiplication is inherently highly parallelizable and involve little control flow. Hence, it's ideal for {\bf graphics processing unit (GPU)} architectures. We're not going to talk much about GPUs in this course, but just think ``matrix multiplication + GPU = good''. If you run vectorized code on a GPU using a framework like TensorFlow or PyTorch, it may run 50 times faster than the CPU version. As it turns out, most of the computation in deep learning is matrix multiplications, which is why it's been such an incredibly good match for GPUs.
\end{itemize}
\end{enumerate}


First, we need to represent the data and model parameters in the form of matrices and vectors. If we have $\ndata$ training examples, each $\nfeat$-dimensional, we will represent the inputs as an $\ndata \times \nfeat$ matrix $\inputMatrix$. Each row of $\inputMatrix$ corresponds to a training example, and each column corresponds to a single input dimension. The weights are represented as a $\nfeat$-dimensional vector $\weights$, and the targets are represented as a $\ndata$-dimensional vector $\targets$.\marginnote{In general, matrices will be denoted with capital boldface, vectors with lowercase boldface, and scalars with plain type.}

The predictions are computed using a matrix-vector product
\begin{equation}
\predictions = \inputMatrix \weights + \bias \onesVec,  \label{eqn:predictions-vec}
\end{equation}
where $\onesVec$ denotes a vector of all ones. We can express the cost function in vectorized form:\marginnote{You should stop now and try to show that these equations are equivalent to Eqns.~\ref{eqn:cost-from-loss}--\ref{eqn:cost}. The only way you get comfortable with this is by practicing.}
\begin{align}
\cost &= \frac{1}{2 \ndata} \| \predictions - \targets \|^2 \\
&= \frac{1}{2 \ndata} \left\| \inputMatrix \weights + \bias \onesVec - \targets \right\|^2.
\end{align}
Note that this is considerably simpler than Eqn.~\ref{eqn:cost}. Even more importantly, it saves us from having to explicitly sum over the indices $\dataIdx$ and $\featIdx$. As our models get more complicated, we would run out of convenient letters to use as indices if we didn't vectorize.

Now let's revisit the exact solution for linear regression. We derived a system of linear equations, with coefficients $A_{\featIdx \featIdxTwo} = \frac{1}{\ndata} \sum_{\dataIdx=1}^\ndata \inputIJ{\dataIdx}{\featIdx} \inputIJ{\dataIdx}{\featIdxTwo}$ and $c_{\featIdx} = \frac{1}{\ndata} \sum_{\dataIdx=1}^\ndata \inputIJ{\dataIdx}{\featIdx} \targetI{\dataIdx}$. In terms of linear algebra, we can write these as the matrix $\mathbf{A} = \frac{1}{\ndata} \inputMatrix^\transpose \inputMatrix$ and $\mathbf{c} = \frac{1}{\ndata} \inputMatrix^\transpose \targets$. The solution to the linear system $\mathbf{A} \weights = \mathbf{c}$ is given by $\weights = \mathbf{A}^{-1} \mathbf{c}$ (assuming $\mathbf{A}$ is invertible), so this gives us a formula for the optimal weights:
\begin{equation}
\weights = \left( \inputMatrix^\transpose \inputMatrix \right)^{-1} \inputMatrix^\transpose \targets.
\end{equation}
An exact solution which we can express with a formula is known as a {\bf closed-form solution}. %This closed-form solution can be implemented with a single line of Python code: \TODO{}

Similarly, we can vectorize the gradient descent update from Eqn.~\ref{eqn:expanded-gd-update}:
\begin{equation}
\weights \gets \weights - \frac{\lrate}{\ndata} \inputMatrix^\transpose (\predictions - \targets),
\end{equation}
where $\predictions$ is computed as in Eqn.~\ref{eqn:predictions-vec}. %This can be implemented with the following very short Python program: \TODO{}

%Contrast this with the naive implementation of the gradient descent updates: \TODO{}

%\TODO{running time comparison}


\section{Feature mappings}

Linear regression might sound pretty limited. What if the true relationship between inputs and targets is nonlinear?  Fortunately, there's an easy way to use linear regression to learn nonlinear dependencies: use a feature mapping. I'll introduce this by way of an example. Suppose we want to approximate it with a cubic polynomial. In other words, we would compute the predictions as:
\begin{equation}
\prediction = \weightJ{3} \inputUni^3 + \weightJ{2} \inputUni^2 + \weightJ{1} \inputUni + \weightJ{0}.
\end{equation}
This setting is known as {\bf polynomial regression}. 

Let's use the squared error loss function, just as with ordinary linear regression. The important thing to notice is that algorithmically, polynomial regression is \emph{no different} from linear regression. We can apply any of the linear regression algorithms described above, using $(x, x^2, x^3)$ as the inputs. Mathematically, we define a feature mapping $\featureMap$, in this case\marginnote{Just as in Section~\ref{sec:partial-derivatives-zero}, we're including a constant feature to account for the bias term, since this simplifies the notation.}
\begin{equation}
\featureMap(\inputUni) = \begin{pmatrix} 1 \\ \inputUni \\ \inputUni^2 \\ \inputUni^3 \end{pmatrix},
\end{equation}
and compute the predictions as $\prediction = \weights^\transpose \featureMap(\inputUni)$ instead of $\weights^\transpose \inputVec$. The rest of the algorithm is completely unchanged. %Figure \TODO{} shows the least-squares cubic polynomial for the data.

Feature maps are a useful tool, but they're not a silver bullet, for several reasons:
\begin{itemize}
\item The features must be known in advance. It's not always easy to pick good features, and up until very recently, feature engineering would take up most of the time and ingenuity in building a practical machine learning system.
\item In high dimensions, the feature representations can get very large. For instance, the number of terms in a cubic polynomial is \emph{cubic} in the dimension!\marginnote{It's possible to work with polynomial feature maps efficiently using something called the ``kernel trick,'' but that's beyond the scope of this course.}
\end{itemize}
In this course, rather than construct feature maps, we will use neural networks to learn nonlinear predictors directly from the raw inputs. In most cases, this eliminates the need for hand-engineering of features.


\section{Generalization}

We don't just want a learning algorithm to make correct predictions on the training examples; we'd like it to {\bf generalize} to examples it hasn't seen before. The average squared error on novel examples is known as the {\bf generalization error}, and we'd like this to be as small as possible.

Returning to the previous example, let's consider three different polynomial models: (a) a linear function, or equivalently, a degree 1 polynomial; (b) a cubic polynomial; (c) a degree-10 polynomial. %\TODO{figure} 
The linear function may be too simplistic to describe the data; this is known as {\bf underfitting}.\marginnote{The terms underfitting and overfitting are a bit misleading, since they suggest the two phenomena are mutually exclusive. In fact, most machine learning models suffer from \emph{both} problems simultaneously.} The degree-10 polynomial may be able to fit every training example exactly, but only by learning a crazy function. It would make silly predictions everywhere except the observed data. This is known as {\bf overfitting}. The cubic polynomial is a reasonable compromise. We need to worry about both underfitting and overfitting in pretty much every application of machine learning.

The degree of the polynomial is an example of a {\bf hyperparameter}. Hyperparameters are values that we can't include in the training procedure itself, but which we need to set using some other means.\marginnote{Statisticians prefer the term \emph{metaparameter} since \emph{hyperparameter} has a different meaning in statistics.} 
%In this example, we chose the degree of the polynomial based on whether the fit looked good visually. This only works for the simplest toy examples. 
In practice, we normally tune hyperparameters by partitioning the dataset into three different subsets:
\begin{enumerate}
\item The {\bf training set} is used to train the model.
\item The {\bf validation set} is used to estimate the generalization error of each hyperparameter setting.
\item The {\bf test set} is used at the very end, to estimate the generalization error of the final model, once all hyperparameters have been chosen.
\end{enumerate}
We will talk about validation and generalization in a lot more detail later on in this course.


\end{document}
















