Review of Linear Regression Models

As you should know, the linear regression model is normally characterized with the following equation:

yi = a + ß1x1 + ß2x2 + ··· ßkxk + ei {or use ß0 for a}.

Consider this equation and try to answer the following questions:

• What does the yi represent? The ß? The x? (Which often include subscripts i — do you remember why?) The ei?

• How do we judge the size and direction of the ß?

• How do we decide which xs are important and which are not? What are some limitations in trying to make this decision?

• Given this equation, what is the difference between prediction and explanation?

• What is this model best suited for?

• What role does the mean of y play in linear regression models?

• Can the model provide causal explanations of social phenomena?

• What are some of its limitations for studying social phenomena and causal processes?

Researchers often use an estimation technique known as ordinary least squares (OLS) to estimate this regression model. OLS seeks to minimize the following:

[MATHEMATICAL EXPRESSION OMITTED]

The SSE is the sum of squared errors, with the observed y and the predicted y (y-hat) utilized in the equation. In an OLS regression model that includes only one explanatory variable, the slope (ß1) is estimated with the following least squares equation:

[MATHEMATICAL EXPRESSION OMITTED]

Notice that the variance of x appears in the denominator, whereas the numerator is part of the formula for the covariance (cov(x, y)). Given the slope, the intercept is simply

[MATHEMATICAL EXPRESSION OMITTED]

Estimation is more complicated in a multiple OLS regression model. If you recall matrix notation, you may have seen this model represented as

Y = Xß + e.

The letters are bolded to represent vectors and matrices, with Y representing a vector of values for the outcome variable, X indicating a matrix of explanatory variables, and ß representing a vector of regression coefficients, including the intercept (ß0) and slopes (ßi). The OLS regression coefficients may be estimated with the following equation:

[??] = (XX)-1 X'Y.

A vector of residuals is then given by

e = Y - Y[??].

Often, the residuals are represented as e to distinguish them from the errors, e. You should recall that residuals play an important role in linear regression analysis. Various types of residuals also have a key role throughout this book. Assuming a sample and that the model includes an intercept, some of the properties of the OLS residuals are (a) they sum to zero ([summation] ei = 0), (b) they have a mean of zero (E]e] = 0), and (c) they are uncorrelated with the predicted values of the outcome variable (r(e,[??])=0).

Analysts often wish to infer something about a target population from the sample. Thus, you may recall that the standard error (SE) of the slope is needed since, in conjunction with the slope, it allows estimation of the ITLt-values and the p-values. These provide the basis for inference in linear regression modeling. The standard error of the slope in a simple OLS regression model is computed as

[MATHEMATICAL EXPRESSION OMITTED]

Assuming we have a multiple OLS regression model, as shown earlier, the standard error formula requires modification:

[MATHEMATICAL EXPRESSION OMITTED]

Consider some of the components in this equation and how they might affect the standard errors. The matrix formulation of the standard errors is based on deriving the variance-covariance matrix of the OLS estimator. A simplified version of its computation is

[MATHEMATICAL EXPRESSION OMITTED]

Note that the numerator in the right-hand-side equation is simply the SSE since (yi - [bar.y] or ei or e1. The right-hand-side equation is called the residual variance or the mean squared error (MSE). You may recognize that it provides an estimate — albeit biased, but consistent — of the variance of the errors. The square roots of the diagonal elements of the variance–covariance matrix yield the standard errors of the regression coefficients. As reviewed subsequently, several of the assumptions of the OLS regression model are related to the accuracy of the standard errors and thus the inferences that can be made to the target population.

OLS results in the smallest value of the SSE, if some of the specific assumptions of the model discussed later are satisfied. If this is the case, the model is said to result in the best linear unbiased estimators (BLUE) (Weisberg, 2013). It is important to note that this says best linear, so we are concerned here with linear estimators (there are also nonlinear estimators). In any event, BLUE implies that the estimators, such as the slopes, from an OLS regression model are unbiased, efficient, and consistent. But what does it mean to say they have these qualities? Unbiasedness refers to whether the mean of the sampling distribution of a statistic equals the parameter it is meant to estimate in the population. For example, is the slope estimated from the sample a good estimate of an analogous slope in the population? Even though we rarely have more than one sample, simulation studies indicate that the mean of the sample slopes from the OLS regression model (if we could take many samples from a population), on average, equals the population slope (see Appendix B). Efficiency refers to how stable a statistic is from one sample to the next. A more efficient statistic has less variability from sample to sample; it is therefore, on average, more precise. Again, if some of the assumptions discussed later are satisfied, OLS-derived estimates are more efficient — they have a smaller sampling variance — than those that might be estimated using other techniques. Finally, consistency refers to whether the statistic converges to the population parameter as the sample size increases. Thus, it combines characteristics of both unbiasedness and efficiency.

A standard way to consider these qualities is with a target from, say, a dartboard. As shown in figure 1.1, estimators from a statistical model can be imagined as trying to hit a target in the population known as a parameter. Estimators can be...