Residuals

Introduction

Key Concepts

Definition of Residuals

In the context of regression analysis, a residual is the difference between an observed value and the value predicted by the regression line. Mathematically, it is expressed as:

$e_i = y_i - \hat{y}_i$

where:

• $e_i$ is the residual for the $i^{th}$ observation.
• $y_i$ is the observed value.
• $\hat{y}_i$ is the predicted value from the regression model.

Purpose of Analyzing Residuals

Analyzing residuals serves several purposes in regression analysis:

• Model Fit Evaluation: Residuals help determine how well the regression model fits the data. Smaller residuals indicate a better fit.
• Assumption Checking: Residual analysis aids in verifying the assumptions of linear regression, such as linearity, homoscedasticity, and normality of residuals.
• Identification of Outliers: Large residuals can signal outliers or anomalies that may influence the regression model disproportionately.
• Detection of Patterns: Systematic patterns in residuals suggest that the model may be missing key variables or that a non-linear model might be more appropriate.

Calculating Residuals

To calculate residuals, follow these steps:

1. Determine the regression equation, typically in the form $y = mx + b$.
2. Use the equation to calculate the predicted value ($\hat{y}$) for each observed value of $x$.
3. Subtract the predicted value from the observed value to obtain the residual ($e$).

For example, consider the regression equation $y = 2x + 3$. If an observed value is $y = 11$ when $x = 4$, the predicted value is:

$\hat{y} = 2(4) + 3 = 11$

Thus, the residual is:

$e = 11 - 11 = 0$

Residual Plots

A residual plot graphs residuals on the vertical axis against the independent variable ($x$) or the predicted values ($\hat{y}$) on the horizontal axis. Residual plots are instrumental in diagnosing the fit of a regression model:

• Random Scatter: A random distribution of residuals around zero suggests that the model's assumptions are met.
• Non-Random Patterns: Patterns such as curves or systematic structures indicate potential issues like non-linearity or heteroscedasticity.
• Increasing or Decreasing Spread: A funnel shape in the residual plot suggests heteroscedasticity, where the variability of residuals changes with the level of $x$.

Types of Residuals

• Raw Residuals: The basic residuals calculated as $e_i = y_i - \hat{y}_i$.
• Standardized Residuals: Residuals divided by their estimated standard deviation, allowing for comparison across different observations.
• Studentized Residuals: Similar to standardized residuals but take into account the influence of each data point on the regression model.

Sum of Residuals

One important property of residuals in linear regression is that their sum is always zero:

$\sum_{i=1}^{n} e_i = 0$

This occurs because the ordinary least squares (OLS) method minimizes the sum of squared residuals, inherently balancing positive and negative residuals.

Coefficient of Residuals

While residuals are individual differences, aggregated measures such as the standard error of the estimate provide information on the average distance that the observed values fall from the regression line. It is calculated as:

$$ SE = \sqrt{\frac{\sum_{i=1}^{n} e_i^2}{n - 2}} $$

where $n$ is the number of observations. A smaller standard error indicates a tighter clustering of points around the regression line, signifying a better fit.

Assumptions Related to Residuals

Analyzing residuals helps verify the key assumptions of linear regression:

• Linearity: The relationship between $x$ and $y$ is linear if residuals do not display a systematic pattern.
• Homoscedasticity: The residuals have constant variance across all levels of $x$.
• Independence: Residuals are independent of each other, meaning the residual for one observation does not predict another.
• Normality: Residuals are normally distributed, which is essential for hypothesis testing and constructing confidence intervals.

Influence of Outliers on Residuals

Outliers, or data points that deviate markedly from others, can significantly impact residuals and the overall regression model:

• Leverage: Outliers with extreme predictor ($x$) values have high leverage and can disproportionately influence the slope of the regression line.
• Influence: Points with large residuals have high influence and can affect the accuracy and reliability of the model.
• Identifying Outliers: Residual analysis helps in detecting these outliers, enabling researchers to make informed decisions about data inclusion.

Reducing Residuals

To minimize residuals and improve model accuracy, consider the following strategies:

• Variable Transformation: Applying transformations (e.g., logarithmic, square root) to variables can address non-linearity and heteroscedasticity.
• Adding Predictor Variables: Including relevant variables that were previously omitted can enhance the model's explanatory power.
• Removing Outliers: Excluding outliers that unduly influence the model can result in a more accurate and reliable regression equation.

Residuals in Multiple Regression

In multiple regression, residuals become even more critical as models include multiple predictors:

• Multicollinearity: High correlation between predictors can inflate residuals and make coefficient estimates unstable.
• Partial Residual Plots: These plots help visualize the relationship between a single predictor and the response variable while accounting for other predictors.
• Adjusted Residuals: Taking into account multiple predictors can lead to more accurate residual analysis in complex models.

Comparison Table

Summary and Key Takeaways