ECON 407: OLS

Equation \eqref{ols:eq:popeq} can also be written using matrix notation. First, define the matrices \(\mathbf{y}\) as

\begin{equation} \mathbf{y}= \begin{bmatrix} y_1 \\ \vdots \\ y_i \\ \vdots \\ y_N \\ \end{bmatrix}_{N \times 1} \end{equation}

Notice that this vector is of dimension \(N \times 1\), or N rows by 1 column. In a similar way, we can construct the matrix of independent variable, \(\mathbf{x}\) as

\begin{equation} \mathbf{x}= \begin{bmatrix} 1 & x_{11} & x_{12} & x_{13} \\ \vdots & \vdots & \vdots & \vdots \\ 1 & x_{i1} & x_{i2} & x_{i3} \\ \vdots & \vdots & \vdots & \vdots \\ 1& x_{N1} & x_{N2} & x_{N3} \\ \end{bmatrix}_{N \times 4} \end{equation}

Using \(\mathbf{y}\) and \(\mathbf{x}\), "stack" each observation and rewrite Equation \eqref{ols:eq:popeq} as

\begin{equation} \begin{bmatrix} y_1 \\ \vdots \\ y_i \\ \vdots \\ y_N \\ \end{bmatrix}_{N \times 1} = \begin{bmatrix} 1 & x_{11} & x_{12} & x_{13} \\ \vdots & \vdots & \vdots & \vdots \\ 1 & x_{i1} & x_{i2} & x_{i3} \\ \vdots & \vdots & \vdots & \vdots \\ 1& x_{N1} & x_{N2} & x_{N3} \\ \end{bmatrix}_{N \times 4} \times \begin{bmatrix} \beta_0 \\ \beta_1 \\ \beta_2 \\ \beta_3 \\ \end{bmatrix}_{4 \times 1} + \begin{bmatrix} \epsilon_1 \\ \vdots \\ \epsilon_i \\ \vdots \\ \epsilon_N \\ \end{bmatrix}_{N \times 1} \end{equation}

Which in compact notation can be written

\begin{equation} \mathbf{y}=\mathbf{x}\mathbf{\beta}+\mathbf{\epsilon} \end{equation}

Using matrix multiplication, we can simplify the middle term as

\begin{equation} \begin{bmatrix} y_1 \\ \vdots \\ y_i \\ \vdots \\ y_N \\ \end{bmatrix}_{N \times 1} = \begin{bmatrix} \beta_0 + \beta_1 x_{11} + \beta_2 x_{12} + \beta_3 x_{13} \\ \vdots \\ \beta_0 + \beta_1 x_{i1} + \beta_2 x_{i2} + \beta_3 x_{i3} \\ \vdots \\ \beta_0 + \beta_1 x_{N1} + \beta_2 x_{N2} + \beta_3 x_{N3} \\ \end{bmatrix}_{N \times 1} + \begin{bmatrix} \epsilon_1 \\ \vdots \\ \epsilon_i \\ \vdots \\ \epsilon_N \\ \end{bmatrix}_{N \times 1} \end{equation}

Further simplifying, we can use matrix addition to find

\begin{equation} \begin{bmatrix} y_1 \\ \vdots \\ y_i \\ \vdots \\ y_N \\ \end{bmatrix}_{N \times 1} = \begin{bmatrix} \beta_0 + \beta_1 x_{11} + \beta_2 x_{12} + \beta_3 x_{13} + \epsilon_1 \\ \vdots \\ \beta_0 + \beta_1 x_{i1} + \beta_2 x_{i2} + \beta_3 x_{i3} + \epsilon_i \\ \vdots \\ \beta_0 + \beta_1 x_{N1} + \beta_2 x_{N2} + \beta_3 x_{N3} +\epsilon_N \\ \end{bmatrix}_{N \times 1} \end{equation}

Notice that for each observation (or row of data), we have the equivalent result as in equation \eqref{ols:eq:popeq}:

\begin{equation} y_i=\beta_0+\beta_1 x_{i1}+\beta_2 x_{i2}+\beta_3 x_{i3} +\epsilon_i \end{equation}

Since the \(\beta\)'s nor the disturbance terms \(\epsilon_i\) in the population equation are observable, we can estimate them (call them \(\mathbf{b}\)). For any value of \(\mathbf{b}\), we can construct the estimated values for the disturbance terms as

\begin{equation} e_i=y_i-(b_0+b_1 x_{i1}+b_2 x_{i2}+b_3 x_{i3}) \end{equation}

Notice that the construction of \(e_i\) is purely an algebraic relationship between the \(\mathbf{b}\) in the estimating equation, the independent variables, and the dependent variables. Consequently, we may want to consider a method that fits this line by choosing \(\mathbf{b}\)'s that in some sense minimizes the distance between the fitted line and the dependent variable.

The OLS Figure shows these various relationships between the population regression line (in blue) and the fitted regression line (in red). Notice, we have not yet said how the \(\mathbf{b}\) are to be estimated yet. Both the population and the sample regression ensure that

\begin{equation} \label{ols:eq:eandeps} y_i \equiv b_0+b_1 x_{i1}+b_2 x_{i2}+b_3 x_{i3}+e_i \equiv \beta_0+\beta_1 x_{i1}+\beta_2 x_{i2}+\beta_3 x_{i3} + \epsilon_i \end{equation}

by definition. Furthermore, notice that the vector \(\mathbf{b}\) is an imprecise measure of the vector \(\beta\). These imprecisions are reflected in the size of the disturbance term (\(\epsilon_i\)) versus the estimated error \(e_i\) in the figure. It need not be the case that \(\mid \epsilon_i \mid< \mid e_i \mid\). For example, consider a point to the left of and above the intersection between the population and the sample regression line. Therefore, for some observations in the sample, the fitted line may have smaller error than the true regression relationship.

Figure 1. Disturbance Terms, Parameters, and Variable in OLS

The least squares is the most commonly used criteria for fitting our sample regression line to the data. Building on our matrix algebra work from earlier, we can derive the least squares estimator. The intuition behind the least squares estimator is that it seeks to minimize the sum of squared errors across the sample in the model. Recalling that the error for the sample equation is

\begin{equation} e_i=y_i-(b_0+b_1 x_{i1}+b_2 x_{i2}+b_3 x_{i3}) \end{equation}

The sum of squares criteria seeks to find an estimated value for each element of \(\mathbf{b}\) that minimizes

\begin{equation} \label{eq:sumofsq1} \sum^{n}_{i=1}(e_i)^2=\sum^{n}_{i=1}(y_i-(b_0+b_1 x_{i1}+b_2 x_{i2}+b_3 x_{i3}))^2 \end{equation}

Denoting the set of independent variables for observation \(i\) as \(\mathbf{x_i}=\begin{bmatrix} 1 & x_{i1} & x_{i2} & x_{i3} \end{bmatrix}_{1 \times 4}\) and the vector

\begin{equation} \mathbf{b}=\begin{bmatrix} b_0 \\ b_1 \\ b_2 \\ b_3 \\ \end{bmatrix}_{4 \times 1} \end{equation}

we can rewrite equation \eqref{eq:sumofsq1} as

\begin{equation} \sum^{n}_{i=1}(e_i)^2=\sum^{n}_{i=1}(y_i-\mathbf{x_i} \mathbf{b} )^2 \end{equation}

Notice that defining the parameter vector as a \(4 \times 1\) column vector is necessary for conformability of the matrix multiplication operation.

We can also write the sum of squares as a matrix multiplication operation.
We are now in a position to write the minimization problem in linear algebra notation. Using the foil method and noting the rules for multiplication of matrices, yields

\begin{eqnarray} \min_b S(b)&=&\mathbf{e}'\mathbf{e}=(\mathbf{y}-\mathbf{xb})'(\mathbf{y}-\mathbf{xb}) \\ &=&\mathbf{y'y}-\mathbf{y'xb}-\mathbf{(xb)'y}+\mathbf{(xb)'xb} \end{eqnarray}

Simplify further and recalling that \((AB)^{\prime}=B^{\prime}A^{\prime}\),

\begin{eqnarray} &=&\mathbf{y'y}-\mathbf{y'xb}-\mathbf{b'x'y}+\mathbf{b'x'xb} \\ &=&\mathbf{y'y}-2\mathbf{y'xb}+\mathbf{b'x'xb} \end{eqnarray}

The necessary conditions for a minimum is [1]:

\begin{equation} \label{ols:eq:foc} \frac{\partial{S}}{\partial{\mathbf{b}}}=-2\mathbf{x'y}+\mathbf{2x'xb}=0 \end{equation}

To obtain this result, one must take note of the rule for matrix derivatives outlined in Green on page 981. Important for this problem is the property that for a matrix \(\mathbf{x}\) and \(\mathbf{y}\) and the product \(M=\mathbf{x}'\mathbf{y}\), \(\frac{\partial{M}}{\partial{\mathbf{y}}}=\mathbf{x}\). Simplifying further, the first order condition from equation \eqref{ols:eq:foc},

\begin{equation} \mathbf{x'y}=\mathbf{x'xb} \end{equation}

Premultiplying both sides by \(\mathbf{(x'x)}^{-1}\) gives

\begin{equation} \mathbf{(x'x)}^{-1}\mathbf{x'y}=\mathbf{(x'x)}^{-1}\mathbf{x'xb} \end{equation}

or

\begin{equation} \mathbf{b}=\mathbf{(x'x)}^{-1}\mathbf{x'y} \end{equation}

which is the ordinary least squares estimator for the population parameters, \(\beta\).

Recall the assumptions in the classic multiple regression model:

• Linearity between \(y_i\) and \(\mathbf{x_i}\)
• There is no linear relationships between the columns in the matrix \(\mathbf{x}\)
• Exogeneity of the independent variables: \(E[\epsilon_i|x_{j1},x_{j2},\ldots,x_{j3}]=0\), for all \(j\). This condition ensures that the independent variables for observation i, or for any other observation are not useful for predicting the disturbance terms in the model.
• Homoskedasticity and no autocorrelation: \(E[\epsilon_i|\epsilon_j] \hspace{.05in}\forall j\in n\) and share the same variance \(\sigma^2\).
• The independent variables may be a mixture of constants and random variables.
• Disturbance terms are normally distributed, and given condition (3) and (4): \(\mathbf{\epsilon} \sim N(0,\sigma^2 I)\)

From Econ 308, you probably recall the discussion about heteroskedasticity. This occurs when the variance of the model error is systematically related to the underlying data in the model. Consider Figure 2. Notice that the slope of the line can be estimated without problem using OLS. However, assuming constant variance for the error term will lead to bad standard errors, because the deviations from the regression line (the errors) get larger as education levels increase.

Figure 2. Heteroskedasticity in the Ordinary Least Squares Model

The companion to this chapter shows how to implement many of these ideas in R and Stata.

---

[1] As pointed out in Greene, one additional condition is necessary to ensure that we have minimized the sum of square distances for our sample. Our first order condition finds either a minimum or maximum sum of squares distances, to ensure a minimum, \(\frac{\partial{S}}{\partial{\mathbf{b}}\partial{\mathbf{b'}}}\) must be a positive definite matrix. This is the matrix analog to ensure that the function is convex with respect to changes in the parameters. Green shows that this is indeed the case so we can safely ignore checking for this condition.

[2] A stronger version of the test, termed White's Test is preferred to the Breusch-Pagan Cook Weissberg test shown here. See the companion document for more information.

[3] It is important to note that Stata scales each element of \(\hat{V}\) by \(\frac{N}{N-k}\) to reduce the bias of the estimated variance covariance matrix for small sample corrections.

[4] Stata scales \(\mathbf{\hat{V}}_{cluster}\) by \(\frac{M}{M-1}\frac{N-1}{N-K}\) for small sample corrections when calculating standard errors.