1 Semi-probabilistic model

11 Probabilistic framework

Let the data be generated by a joint distribution of two random variables: a feature vector and a class label from a given parametric family ( is the parameter), with each observation being independent:

Given a training sample, we need to estimate the parameter values and build a predictive model:

12 Model

We will use the formalism of semi-probabilistic models, i.e., we will consider values as fixed and values as variables in the model. In other words, as a predictive model, we will build a probability distribution for the random variable , parameterized by the data and the distribution shape parameters

13 Likelihood function

Let’s write the joint distribution for all observed points as a product of independent distributions:

Applying the logarithm to the likelihood function, we obtain the log-likelihood function:

14 Parameter estimation

Thus, we can find the parameter values and use them in the conditional distribution for prediction:

The prior distribution is canceled out from the product above.

• The maximum likelihood method considers the prior distribution of unknown and

unimportant (unlike in MAP), focusing solely on the conditional distribution which serves as a model for building the algorithm .

• Generally, when the chosen prior distribution is parameter-independent (),

it naturally cancels out since it remains constant for fixed and does not depend on .

15 Cross-entropy loss

From summing over specific points , we can transition to summing over all possible values , assuming zero probability for point belonging to another class and defining by definition.

This is cross-entropy loss, which can be used if the model predicts probabilities of belonging to each class.