In today’s article, we will extend our knowledge of the Simple Linear Regression Model to the case where there are more than one explanatory variables. Under certain conditions, the Gauss Markov Theorem assures us that through the Ordinary Least Squares (OLS) method of estimating parameters, our regression coefficients are the Best Linear Unbiased Estimates, or BLUE (Wooldridge 101). However, if these underlying assumptions are violated, there are undesirable implications to the usage of OLS.
In practice, it is almost impossible to find two economic variables that share a perfect relationship captured by the Simple Linear Regression Model. For example, suppose we are interested in measuring wage for different people in Canada. While it is plausible to assume that education is a valid explanatory variable, most people would agree it is certainly not the only one. Indeed, one may include work experience (in years), age, gender or perhaps even location as regressors.
As such, suppose we have collected the data for multiple variables, x1,… xn, and y. Through a Multiple Linear Regression Model, we can estimate the relationship between y and the various regressors, x1,… xn (Wooldridge 71).
- yi is the ith observation for the independent variable
- xki is the ith observation for the kth regressor
- βk is the coefficient for the kth regressor
- εi is the error term
As in the simple case, we can use the Ordinary Least Squares method (OLS) to derive the estimates for our coefficients in the Multiple Linear Regression Model. Recall, our goal is to summarize the sum of squared residuals, that is (Wooldridge 73) :
If we take the partial derivatives of the above equation with respect to β0, β1, …, βn and set them to zero, the result is a system of n+1 equations. The solution to this system will produce the estimates for each βi.
In general, the OLS method for estimation is preferred because it is easy to use and understand. However, simplicity comes with its limitations. Ordinary Least Squares provides us with a linear estimator of parameters in Multiple Linear Regression. In other words, we obtain a column vector of estimates for βi that can be expressed as a linear function of the dependent variable y. Like all other linear estimators, the ultimate goal of OLS is to obtain the BLUE Let us first agree on a formal definition of BLUE. On one hand, the term “best” means that it has “lowest variance”; on the other, unbiasedness refers to the expected value of the estimator being equivalent to the true value of the parameter (Wooldridge 102).
We now turn our attention to the Gauss Markov Theorem, which guarantees that the Ordinary Least Squares method under certain conditions. They are colloquially referred to as the Gauss Markov Assumptions. It is important to note that the first four ensure the unbiasedness of the linear estimator, while the last one preserves the lowest variance (Wooldridge 105).
- Linearity in Parameters
- Random Sampling
- No Perfect Collinearity
The first two assumptions are self-explanatory; the parameters we are estimating must be linear, and our sample data is to be collected through a randomized, probabilistic mechanism. The third condition, no perfect collinearity, ensures that the regressors are not perfectly correlated with one another. An example of this is including both outcomes of a binary variable into a model. Suppose we are interested in official language preferences: if we were to add English and French as regressors, the model would exhibit perfect collinearity because we know if someone prefers English, they do not prefer French at the exact same time. Mathematically, if they were both indicator variables, we would not be able to differentiate when an observation prefers English or French because one of them will always have a value of 1. Exogeneity means that the regressors cannot be correlated with the error term. The converse of this is endogeneity, and examples of this include omitted variable bias, reverse causality, and measurement error. The fifth and final assumption is homoscedasticity, which means the variance of the error term must be constant no matter what the value of regressors are.
Admittedly, no one will ever walk up to you and ask “What are the conditions for the Gauss Markov Theorem?”. However, as the first article alluded to a few weeks ago, we need to use econometric models with discretion. To put the importance of these assumptions into perspective, consider this analogy. The criminal code is in place so that the citizens of our country can function well together without harming one another. A police officer will never come up to you and ask you to recite the criminal code, but when you start violating the laws, you will likely find yourself in trouble. It is important for us to identify when we are breaking the law, and find methods to avoid doing so. The same can be said using OLS. By learning the five assumptions, we know of possible issues that we may run into when performing linear regression.
In summary, let’s end the discussion of OLS with more insights on the Gauss Markov Theorem. If all of the conditions simultaneously hold, we know that OLS can is BLUE. In later articles, we will discuss specific ways to mitigate violations of these conditions. For example, when we have endogeneity present (the fourth assumption is violated), our OLS estimator will be biased. We will talk about methods to solve this issue like performing an Instrumental Variable Estimation to produce unbiased estimates.
- Wooldridge, Jeffrey M. Introductory Econometrics: A Modern Approach. 5th ed. Mason, OH: South-Western Cengage Learning, 2013. Print.