1 Canonical form (1D)
11 Canonical form 1
The exponential family represents a parametric class of probability distributions defined by their probability density function (pdf) or probability mass function (pmf):
where represents a value of random variable , is a parameter, represents a parameter-dependent normalization constant, and is a parameter-independent scaling function, also called the carrier measure. In short notation, This family encompasses many common probability distributions. Any distribution whose pdf can be expressed in the form of (1) belongs to the exponential family.
The equation (1) is the canonical form of the exponential family. The canonical form provides a standardized way to express all exponential and pre-exponential terms.
12 Partition function
To hold normalization, the term called the partition function is introduced:
Corresponding logarithm is called the log partition function or cumulant function.
13 Sufficient statistics
If the random variable does not have a linear relationship with the parameter , a function called sufficient statistics is introduced to make the relationship linear: Technically, is a new random variable for which (1) holds.
14 Canonical form 2
Equivalently to (3), the exponential family can be rewritten as a single exponential function when all pre-exponential terms are gathered:
where is the log-partition (cumulant) function, and scales the distribution. Both forms are canonical as they are equivalent.
15 Fitting parameter
For a data points , we can estimate by standard approaches, e.g. by maximizing the likelihood function:
the terms are constant and can be ignored.
16 Modeling
While 1D exponential family can be used to model 1D densities, relationships between two variables and still can be modeled. If we assume that has an exponential family distribution , and joint distribution is in the form of : Assumptions: Distributions between and are independent (inputs do not depend on the parameter), the distribution of is assumed uniform (data density is constant); as a result, the distribution of answers is conditioned both by the input and the parameter of the exponential family.
If can be expressed as , then .
2 Canonical form (D)
21 Vector parameter
The scalar parameter combines with sufficient statistics to produce a scalar value within the exponential function . Parameters are linear, i.e. they linearly transform the random vector (or its sufficient statistics ) to produce the scalar value.
Generalizing to a vector requires only that the inner product exists, where maps the random variable into the same space where resides:
22 Random vector
The generalization from scalar random variable to random vector follows naturally through sufficient statistics that ensures the inner product exists. While dimensions of and need not match, the sufficient statistics must create a valid inner product .
23 Canonical form
A random vector follows the exponential family distribution with parameter when its pdf takes the form: These equivalent canonical forms relate through and .
The remaining terms generalize naturally:
Partition function:
where represents the differential volume element.
Log partition function:
Carrier measure and its logarithm must be defined for vector argument:
24 Modeling scalar response
Given a joint distribution of -dimensional inputs and scalar responses , we can model their relationship analogous to (6): Note that is a scalar random variable, with only the parameters being vectors: .
For training data , we estimate by maximizing:
For prediction on new data , we calculate the conditional expectation:
25 Modeling all responses
A straightforward approach collects all responses for into a column vector . The notation indicates each training example shares a common parameter .
Vector can be represented as a vector of all answers :
26 Modeling a single vector response
For vector-valued responses, each represents multiple outputs for a single input . The joint distribution follows:
Different responses can be modeled with a shared vector or individual parameters:
3 Bernoulli distribution
31 Classical definition
Suppose we have a scenario with two outcomes: “success” and “failure,” represented by a binary random variable . The probability of “success” is defined by a parameter .
In short, means that follows the Bernoulli distribution with parameter . The probability mass function (pmf) is:
The Bernoulli distribution is perhaps the simplest member of the exponential family.
32 Problem statement
To express the Bernoulli distribution, we need to explicitly identify all components of the exponential family’s pdf/pmf: parameter , sufficient statistic , partition function , and scaling function .
33 Canonical form
Starting by taking the logarithm of the classical definition:
Undoing the logarithm, we get: Our goal is to demonstrate that this us equivalent to the canonical form (8):
By comparing this with the canonical pmf (8), we can easily identify:
34 Logit function
The term “logit” is a variation of “logarithm” as it comprises the logarithm function. You can think of it as a portmanteau of “logarithm” and “unit.” The parameter of the exponential family distribution depends on the parameter of the classical Bernoulli distribution . This connection is established by the logit function: The relation of the probability of an event to the probability of the complementary event is called the odds ratio: The logit function is the logarithm of the odds ratio:
In other words, the canonical parameter can be easily calculated as . Technically, the logit function maps the probability to the arbitrary real number as the logarithm of can be any real number.
35 Sigmoid function
Likewise, the classical probability can be easily calculated from the canonical parameter by applying an inverse function to the logit function:
The commonly known sigmoid function is the inverse of the logit function. Inverse of logit: The inverse of the logit function is the sigmoid function:
36 Partition function
The pre-exponential term is equal to , as we have shown. By applying , we can see that the pre-exponential term depends only on the canonical parameter , not on the input , so
37 Final form
The Bernoulli distribution in canonical exponential family form is:
4 Normal distribution
41 Standard normal distribution
In the trivial case, the standard normal distribution can be expressed as an exponential family distribution :
Note: The choice of , , , and is not unique. where the coefficient before is the canonical parameter , the sufficient statistics are , and the pre-exponential term is .
42 Non-standard normal distribution
Interestingly, the non-standard normal distribution cannot be easily fitted into the exponential family. The pdf of the non-standard normal distribution is:
The sufficient statistics and canonical parameter are both 2D vectors:
Note: Since , the canonical parameter must be negative; this constrains the parameter space.
The parameters of the original distribution and can be expressed as:
The partition function depends on two parameters, and the scaling function is a constant, so the pre-exponential term is:
Finally, the canonical form of the non-standard normal distribution is:
43 Multivariate normal distribution
Further generalization is relatively straightforward; the pdf of the multivariate normal distribution is:
where is a random vector, is the mean vector, and is the covariance matrix, the sufficient statistics, canonical parameters and pre-exponential term are:
5 Laplace distribution
51 Classical definition
The Laplace distribution arises naturally as the difference between two independent, identically distributed exponential variables. For this reason, it is also called the double exponential distribution.
The distribution has two parameters: is the location parameter (mean), and is the scale parameter. Its pdf is similar to the normal distribution but has an absolute value in the exponent instead of a square:
This distribution is useful for modeling data with sharp peaks and heavy tails compared to the normal distribution.
52 Special case
When , the Laplace distribution can be expressed in exponential family form:
The canonical parameter becomes , the sufficient statistics , and the partition function .
53 General case
For , the Laplace distribution cannot be written as an exponential family distribution because cannot be represented as sufficient statistics , which by definition must be independent of distribution parameters.
The classical Laplace distribution parameters behave differently. The parameter directly relates to the canonical parameter through in the exponential family form. However, the parameter does not correspond to any canonical parameter, making it impossible to express the doubly-parameterized Laplace distribution in exponential family form.
54 Trick 1: Shifting by
By shifting the distribution by and introducing a new variable , the distribution of follows the exponential family form:
Thus, while the general Laplace distribution itself lies outside the exponential family, the distribution of the shifted variable belongs to it.
55 Trick 2: Fixing
Alternatively, fixing to any constant value allows defining sufficient statistics , which expresses the Laplace distribution in exponential family form:
6 Expectation and mean parameter
61 Expectation of sufficient statistic
Consider an exponential family distribution:
Starting with the probability normalization condition:
Taking the gradient of both sides:
Switching the order of the gradient and integral operators:
Computing the gradient of the exponential function:
Rearranging terms:
Therefore, the expectation of the sufficient statistic is:
62 Expectation of Bernoulli distribution
The pmf of a Bernoulli random variable is:
The corresponding expectation can be calculated via differentiation:
Derivative of :
As shown in “Bernoulli distribution”, the classical parameter .
63 Expectation of Poisson distribution
64 Connection to link function
This result directly connects to the mean parameter and link function:
7 Poisson distribution
71 Classical definition
The Poisson distribution models the number of events occurring within a fixed interval of time (or space). The distribution has a single parameter representing the average rate of event occurrences.
The pmf of the Poisson distribution is:
is the set of non-negative integers.
where represents the number of events occurring in the interval. This pmf can be rewritten in exponential family form.
72 Solution
To express (47) as a one-dimensional exponential family distribution:
we combine all parameter-dependent terms ( and ) from (47) into a single exponent, and gather all parameter-independent terms () into the pre-exponential term:
The relationship between classical parameter and canonical parameter is given by or equivalently
The log partition function follows from (49) and the relationship (see the previous note).
Comparing terms with the canonical form yields the canonical parameter , the log partition function , and the scaling function .
73 Mean parameter
The expectation follows directly from the derivative of the log partition function:
obtained through the formalism of the exponential family.
74 Classical approach
The same result emerges by directly calculating the expectation using the classical pmf:
The exponential function expands as a Taylor series:
The summation index changes twice: first to factor out from , and then through the substitution .
As shown, since , the mean parameter . This demonstrates that the mean parameter of the exponential form directly corresponds to the classical expectation.