1 General concept of regularization

The regularization method traces back to A.N. Tikhonov’s work in 1963, who proposed it for solving ill-posed problems where formal mathematical solutions are meaningless.

A linear system has no solution when is singular (), rank-deficient (), or when data contains errors preventing from being satisfied:

Adding constraints on model parameters through a regularizer term shrinks the solution space, making it possible to find (sometimes) a practically useful approximate solution:

Regularization is a technique that imposes constraints on model parameters to control the solution space. By limiting where parameters can be found, it effectively shrinks the space of possible solutions. This reduction in parameter flexibility not only enhances model generalizability but also helps prevent overfitting by focusing on simpler solutions.

The term derives from Latin “regula” (rule) and “regularis” (in accordance with rules), reflecting its role in establishing systematic constraints on model behavior.

Through these controlled parameter constraints and reduced solution space, regularization helps create simpler, more robust models by reducing their sensitivity to noise in the training data.

2 Probabilistic interpretation of regularization

21 Probabilistic framework

Consider a joint distribution of data and model’s parameters :

  1. The prior distribution of is independent of parameters and can assumed to be uniform and ignored in the model:

  2. The posterior distribution of responses depends on parameters and specific data point , following a semi-probabilistic model formalism. The model is specified by defining the conditional distribution of responses given a specific and model parameters . When the parameters are fitted, we make predictions for new data points by maximizing the probability of a response given :

    Support of a random variable is the set of all possible values that can take with non-zero probability:

  3. The prior distribution of parameters is assumed to be known and defined by the hyperparameter vector :

22 Applying MAP

The joint distribution of data and parameters can be rewritten as a product of conditional pdfs:

We omit random variables in the pdf’s underscripts for brevity. Just look at the arguments before the bar to understand to which random variable the pdf refers: e.g., means .

As it was mentioned, the canceled prior distribution of data is independent of the model parameters . We ignore it (or assume uniform).

Still, we didn’t ignore the prior distribution of parameters , which is . Because of that, it’s MAP (Maximum a Posteriori) estimation, not MLE (Maximum Likelihood Estimation).

A pdf becomes a likelihood function when we consider it as a function of arguments behind the bar, e.g.

23 Finding parameters

For specific training samples and predefined hyperparameters , we write the joint likelihood of data and model parameters and maximize it:

The second term is the regularizer, its strength is defined by constant and hyperparameters . Regularizer narrows the space in which the parameters can be found. The more narrow the space, the more constrained the model is.

After finding the parameter vector estimate , predictions for a new data point can be made by substituting the estimate into the model :

24 Loss-function

Probabilistic regularizer (6) can be rewritten as the empirical risk where it becomes an additional loss function:

3 -norm regularization (Tikhonov regularization)

On a privious step, we found the general form of the regularazer assuming prior distribution of parameters :

31 Model:

Here we make specific choices of the prior distributions parameters and the data:

  1. All parameters are independent and linear, so the joint distribution is a product of the individual distributions: These distributions impose prior constraints on the model coefficients, effectively reducing the solution space

  2. Each parameter follows a Gaussian distribution with two hyperparameters common for all individual distributions: mean and standard deviation :

  3. The data is generated by a linear model with Gaussian noise : Both errors and model’s parameters follow multivariate Gaussian distributions with zero mean and different covariance matrices and respectively.

    Dimensions of vectors and are different:

    All corresponding distribution parameters have appropriate dimensions:

32 Applying MAP

Here we denote pdf underscripts with letters corresponding to random variables For any arbitrary model we can estimate the error term as difference between the predicted and the actual responses. The posterior distribution of parameters :

As is independent of , we cancel it out:

33 Independence

We applied MAP and wrote the optimization problem, now continue with substituting the specific distributions:

The error estimates are directly related to the data:

Let’s write the prior distribution of parameters:

The data distribution can be written through the error distribution:

Let’s write the posterior distribution of parameters:

Let’s write the likelihood function (log-loss):

Let’s rewrite it as empirical risk minimization:

In regularization, everything is similar, but the errors are described by the Laplace distribution.

4 -norm regularization

Unlike regularization, LASSO assumes model errors follow the Laplace distribution, characterized by heavy tails and a sharp peak:

Using MAP for parameter estimation:

Let’s write the likelihood function (log-loss):

Let’s rewrite it as empirical risk minimization:

5 regularization

6 Geometric interpretation of regularization

7 Ridge, LASSO, and Elastic Net regression

References