1 GLM: Logistic regression

Logistic regression: A model predicts if the patient is a vegan () or not () by a result of cholesterol test and gets the result mmol/L. A binary response modeled with Bernoulli distribution , where is the probability of being a vegan.

Bernoulli distribution belongs to the exponential family; in canonical form, it is:

where is the logit function.

Fitting the parameter on the historical data:

Then, to make a prognosis, the probability of being a vegan is calculated:

In logistic regression, the connection between the input and the probability is modeled as:

The inverse of the logit function is the sigmoid function :

The parameters and are trained on the historical data:

Then, to make a prognosis, the probability of a bad outcome is calculated:

NB: We used distribution of to derive the loss function:

but we ignore the distribution of when we make a prediction, we are only interested in

Exponential family: Bernoulli distribution belongs to the exponential family, the exponential family parameter can be calculated from the Bernoulli parameter , then

2 Generalized Linear Models (GLM)

21 Introduction to GLM

A generalized linear model (GLM) extends ordinary linear regression by allowing for response variables that follow any exponential family distribution. The general form is:

22 Making Predictions

To make a prediction in GLM, we estimate the conditional expectation (canonical mean parameter):

For most cases, the sufficient statistics T is trivial, and we can obtain the needed expectation from the distribution parameters:

For example, in logistic regression, the mean parameter corresponds to probability:

23 Mean and Link Functions

231 Mean Function

The mean function describes the expected value of the response variable Y (or sufficient statistics T(Y)) given current parameters:

232 Link Function

The link function connects linear parameters ฮธ = Xฮฒ (linear predictor) to the expected value (canonical mean):

Its inverse calculates parameters:

24 GLM as Linear + Nonlinear Transforms

GLM combines linear and nonlinear transformations:

  1. Linear predictor computation:
  1. Link function application:
  1. Final prediction via inverse link function:

25 Logistic Regression as GLM

Logistic regression is a special case of GLM using Bernoulli distribution:

In canonical form:

The link function can be found from:

Where:

3 GLM: Cross-entropy and log-loss

31 Model

Logistic regression represents a special case of GLM where the binary response variable follows a Bernoulli distribution:

Here, represents the success probability in a single trial. The canonical form of the Bernoulli distribution is:

Starting from the general GLM form:

We can derive both cross-entropy and log-loss directly, assuming only the Bernoulli distribution of .

32 Link Function

The link function connects the response variableโ€™s mean to the distributionโ€™s canonical parameters :

In GLM, we assume the canonical parameters are linear:

where represents the linear coefficients corresponding to features in .

For the Bernoulli distribution, the link function takes the form:

33 Cross-entropy Loss

We begin with the log-likelihood function for the Bernoulli-distributed response variable , assuming :

34 Log-loss

The log-loss function can be derived by taking the negative log-likelihood:

35 Making Predictions

To make a prediction:

References