Gaussian linear regression · Intermediate ML

1 Weighted Least Squares (WLS)

11 Intro

Heteroscedasticity: can be eliminated by applying weighted LS.

For a model with non-constant variance of the error term:

To apply WLS, the weights must have a negative square unit:

This leads to the transformations:

The weight matrix is:

and

Now, the model can be formulated as a homoscedastic least squares problem:

The Weighted Least Squares (WLS) method extends ordinary least squares by incorporating observation-specific weights. The basic model structure remains similar to OLS:

where is the vector of features, is the vector of parameters, is the target variable, and is the error term.

Each observation has associated weights that reflect the importance of that particular observation.
This method minimizes the weighted sum of squared residuals:

The solution to this minimization problem is given by:

where is the diagonal matrix of weights, and is the weighted pseudo-inverse.

12 Weight matrix

For a weighted

let’s introduce the weight matrix:

13 Matrix form

Quadratic form: is a function of the form:

Coefficients can be arranged in a symmetric matrix , and the quadratic form can be written in matrix form as:

Thus, we can rewrite the RSS in matrix form as a quadratic form:

14 Back to standard LS

The weighted LS problem can be easily reformulated as a standard LS problem by replacing the original variables with transformed ones:

Substituting these transformations into the original model, we get:

15 Analytical solution

Now, let’s solve for in the transformed model. Since and are diagonal matrices, transposing them results in the same matrix:

Expanding the expressions: