1 Quantile of a random variable
11 Quantile of a sample
Given an unordered sample , we can construct a sorted sample , where is the th smallest value of the sample (), also known as the th order statistic.
Some order statistics:
NB: In , values are integers, while in , values are fractional, e.g., refers to the 10th order statistic, while refers to the 0.1-quantile.
Informally, the -quantile is the value that divides the ordered sample into two parts with proportions . However, this definition is ambiguous. One practical approach is to use different formulas for and :
In practice, other definitions of quantiles are also used, e.g., for any
12 Quantile of a random variable
For a random variable , the quantile function, denoted either as or , is defined as the inverse of its CDF:
where denotes the infimum, which is the greatest lower bound, i.e., is the smallest value for which the probability is at least .
Example: For a uniform distribution on interval , the CDF is:
The corresponding quantile function is:
For a continuous random variable , the probability of being less than or equal to is given by the cumulative distribution function (CDF): where is the probability density function (PDF).
For a discrete random variable , the CDF is defined as: where is the probability mass function (PMF).
Some important quantiles: Percentiles are also quantiles, e.g. is the 95th percentile.
13 Quantile and probability
CDF maps real numbers to probabilities :
Quantile , being the inverse of CDF, maps probabilities to real numbers :
We specifically denote probability as to emphasize its connection to quantiles.
Technically, is a function of , and it is usually denoted as , similar to PDF and CDF . However, the notation is used here to emphasize the analogy between quantiles and expectation .
The meaning of is that it is the value of such that the probability of being less than or equal to is :
14 Conditional quantile
The generalization of the quantile to the conditional case is straightforward; it’s defined as the inverse of the conditional CDF:
where is the conditional CDF defined via the conditional PDF (continuous case) or PMF (discrete case).
The meaning of the conditional quantile is that it is the value of such that the probability of being less than or equal to given is :
2 Quantile loss
21 Check-loss
Consider an asymmetric loss function parameterized by : This loss function is also called the pinball loss and quantile loss
Strictly speaking, this is an estimation of the error: ; for different estimations of , there are different
where is the error term (residual) and is the prediction of a regression model.
Figure 2: Check loss Figure 3: Check loss Figure 4: Check loss
22 Constant model
Let’s first consider the simplest case, where we look for in the family of all constant models .
For a pair taken from the joint distribution , a function that minimizes can be found by minimizing :
The empirical risk (expected check-loss) can be expressed as:
23 Risk minimization
The integral is split at into two independent regions: and . By differentiating both integrals with respect to , we can find :
At the extreme point , the derivative of the risk is zero:
Thus, the optimal constant model is the -quantile of the random variable :
24 Implications
We assumed that is a constant function of and derived the optimal constant model that minimizes the empirical risk (expected check-loss) . Notably, if we differentiate with respect to any general function , the result remains the same.
Minimizing the check loss for a regression model is equivalent to finding the -quantile of the random variable . Therefore, the algorithm derived from solving the minimization problem effectively predicts the -quantile of . Some implications of minimizing (8):
25 Quantile parameter
By using the check loss , we can train a regression model that predicts the -quantile of the random variable given the input :
where depends both on hyperparameter and on the input . This means that predictions are different for different values of .
Likewise, the error term (residual) depends on :
and the check loss in (8) is actually .
3 Expectation and median
31 Minimization of MSE
The expectation is the average value of a random variable . It can be found by minimizing quadratic loss (MSE):
Differentiating the quadratic loss with respect to gives: Rearranging gives:
which holds for both conditional and unconditional expectations.
The algorithm that minimizes the average quadratic loss has the lowest MSE among all possible estimators and sometimes is called the minimum mean squared error (MMSE) estimator, which is more commonly known as the least squares (LS) estimator.
In other words, minimization of quadratic loss is one of (many) possible ways to find a good model . During training, this model learns how to predict conditional expectation for a given , then we use it to predict the expectation for previously unseen data points .
This estimator has good theoretical guarantees (e.g., unbiasedness, minimum variance, etc. under certain conditions) and because of that is the first choice for most regression problems.
32 Minimization of MAE
An alternative estimator is obtained when instead of minimizing the quadratic term , we replace it with absolute difference , which is equivalent to minimizing mean absolute error (MAE). Interestingly, this gives us the median of the random variable :
Given training data , the empirical estimation according to (15) and (16) can be expressed as:
Indeed, MAE is directly connected to the quantile loss . For , the quantile loss is simply the absolute value of the error (we ignore factor):
This model is also referred to as the Least Absolute Deviations (LAD) estimator.
For , the quantile corresponds to the value such that ; i.e., the value cuts the distribution of in half. This is what the median () of a random variable is.
Figure 5: PDF of standard Normal and Laplace distributions. Laplace has heavier tails.
33 Laplace distribution
Quadratic loss is derived from assuming a Gaussian distribution of . Formally, absolute loss comes from assuming a Laplace distribution of :
Median is more robust to outliers than mean: In an ordered sample , adding an outlier shifts the mean proportionally to its magnitude: In the worst case an extreme outlier or can only shift the median to the adjacent element:
where is the scale parameter and is the mean. As the Laplace distribution is symmetric, the mean is equal to the median .
The Laplacian distribution’s heavier tails (Figure 5) assign higher probabilities to extreme values, making models based on it more robust to outliers.
34 Likelihood
Assuming observations are i.i.d. and algorithm predicts the conditional mean , the likelihood function is given by:
Maximizing the likelihood function is equivalent to minimizing MAE:
35 Measures of central tendency
Expectation and median are two distinct measures of central tendency for a random variable . In some cases, they are equal, but in general they are not. This distinction leads to two different regression models:
Quantile regression is not limited to ; we can construct a regression model for any conditional quantile where is a hyperparameter
- In ordinary least squares (LS), we predict the expected value of a random variable:
- In median regression, we build a model that predicts the conditional median:
4 Quantile regression
Quantile regression was introduced by Roger Koenker and Gilbert Bassett in (Koenker & Bassett, 1978). For a short overview and examples see (Koenker & Hallock, 2001) and (Koenker, 2005) for details.
41 Probabilistic model
Suppose the distribution of the data is modeled as a joint distribution . Our goal is to predict the quantile for a given , i.e., to predict the conditional quantile .
42 Optimization problem
The empirical risk is defined as the average quantile loss (8) over the distribution . By minimizing the empirical risk, we can find the optimal model that predicts the quantile :
43 Practical reformulation
From the theoretical expression of the empirical risk, we can derive a practical reformulation of the quantile regression problem.
In LS regression, only one prediction exists, with a single residual . In quantile regression, is parameterized by , producing multiple possible predictions for the same random variable, with corresponding residuals
For a specific pair drawn from the joint distribution represented by a training set , the empirical risk can be expressed via the check loss (8):
The model can be any general regression model supporting custom loss functions or the quantile loss specifically.
44 Linear quantile regression
The conditional quantile can be modeled as a linear function of predictors :
Figure 6: Linear quantile regression for non-normaly distributed noise
where is a vector of regression coefficients, and are regression coefficients for the feature and a predefined hyperparameter . Coefficients are estimated by minimizing the empirical risk:
45 Neural quantile regression
Neural networks inherently support custom loss functions and can model conditional quantiles as well (Figure 7). A model predicting conditional quantiles must be trained with a quantile loss, which can be easily implemented:
Figure 7: Quantile regression performed by a neural network
class QuantileLoss(L.LightningModule):
def __init__(self, q: float):
super().__init__()
self.q = q
def forward(self, y_pred, y_true):
epsilon = y_true - y_pred
return T.where(
epsilon >= 0,
self.q * epsilon,
(self.q - 1) * epsilon,
).mean()47 Ensemble models
Multiple base algorithms can be combined to create an ensemble model
If each base algorithm is trained to predict quantiles , the ensemble will estimate the expectation of the quantile .
5 Convergence and reliability of quantile regression parameters
51 Linear quantile regression
For linear quantile regression (25), the conditional quantile is modeled as a linear function of predictors . The theoretical properties of linear quantile parameters such as convergence and variance can be derived, though the analysis is more complex than for traditional Gaussian regression.
52 Parameter expectation
All regression coefficients are functions of . Under appropriate conditions (independent observations with finite second moments), the asymptotic distribution of the quantile regression estimator is unbiased:
i.e., theoretically the estimator converges to the expected value of the parameter as the sample size approaches infinity:
53 Parameter variance
The estimator is asymptotically normally distributed with varianceand mean according to (30)
The variance in (31) depends on three terms:
The 1st multiplier determines the convergence rate of the estimator as a function of the sample size ; the larger the sample size, the smaller the variance.
The 2nd multiplier depends on the quantile . As approaches the tails (0 or 1), this term decreases, which would seemingly lower the variance. It reduces variance if isolated, however, this is not the primary contributor to overall variance.
Figure 9: term in (31) reaches its maximum at
The 3rd multiplier is the sandwich variance estimator, which depends on both the estimated parameters and the robust variance matrix . Typical formulations include:
While decreases near the tails, the sandwich term becomes poorly estimated and tends to dominate.
Consequently, the variance of estimated parameters increases as approaches 0 or 1. In practice, predictions near the median are typically more precise, while predictions for extreme quantiles (e.g., 0.01 or 0.99) are less reliable.
54 Bad statistical guarantee
While ordinary least squares (OLS) estimates benefit from the Gauss-Markov theorem, which establishes OLS as the best linear unbiased estimator (BLUE) under classical assumptions, quantile regression follows different asymptotic properties.
The variance of the quantile regression estimator is larger than that of OLS, especially for extreme quantiles
Quantile regression estimators remain unbiased and consistent, but their variance behavior is more complex. As shown in equation (31), the variance depends on both the quantile level and the underlying data distribution through the sandwich estimator term .
In practice, quantile regression estimates exhibit higher statistical variability than OLS estimates, particularly for extreme quantiles (e.g., or ). This occurs because:
- The sparsity of data in the tails leads to less reliable sandwich term estimation
- The conditional density at extreme quantiles becomes more difficult to estimate accurately
- The effective sample size for determining extreme quantiles is effectively reduced
This statistical efficiency trade-off is a necessary cost of gaining robustness to outliers and insights into the complete conditional distribution.
6 Robustness of quantile regression
61 Non-normality (skew, heavy tails, multimodality)
Quantile regression models conditional quantiles, capturing skewed or heavy-tailed distributions without relying on normality assumptions. OLS assumes normality and may produce misleading results when this assumption is violated.
Given uniform , look (Figure 10) at three different models with non-normal additive noises: , , and .
62 Heteroscedasticity
Quantile regression does not assume homoscedasticity (constant variance). Instead, it models different parts of the conditional distribution independently, allowing for varying spread (e.g., wider or narrower intervals) across predictors. OLS assumes homoscedasticity (or equal weight of all observations).
63 Robustness to outliers and noise
By focusing on quantiles rather than the mean, quantile regression reduces sensitivity to random noise and outliers, emphasizing specific distributional trends. Quantile regression also does not assume any specific noise distribution. In OLS, a few outliers can have a pronounced effect on parameter estimates.
64 Censoring
Censoring arises when the response variable is not fully observed. For instance, in clinical trials, the exact value of may be unavailable for some patients. If a patient exits a longitudinal study, we only know they survived up to time , but their true survival time might be much longer (Figure 11).
Figure 11: Time-to-death plot from the start of a clinical trial. Circles represent patients whose exact time-to-death is known, while crosses represent patients who withdrew from the study.
In standard Gaussian regression, censoring results in a bias in estimates since observations are truncated. Quantile regression allows to model different quantiles of the distribution: some areas of the distribution may not be affected by censoring, while others are. By choosing appropriate quantiles, we can obtain reliable estimates even in the presence of censoring.
However, in general it’s better to experiment with different losses and other models, that work with censoring implicitly.
65 Invariance
Quantile regression is invariant to monotonic transformations of like logarithm or square root. In OLS this is not the case, although transformations are sometimes used to normalize data.
7 Interpretation of linear quantile coefficients
71 Impact on target variable
Quantile regression coefficients represent the impact of a unit change in predictor on the response variable at specific quantiles. Unlike OLS coefficients, they capture how features influence different parts of the target distribution.
By examining how varies across quantile levels, we can guess how predictors affect various segments of the conditional distribution, revealing effects that are not directly observable in standard regression.
72 Data
The ACTG 320 clinical trial, initiated in 1997 by Merck, was designed to evaluate the effectiveness of the antiretroviral drug indinavir when used in a triple-drug regimen compared to a standard two-drug treatment for HIV patients.
Table 1: ACTG 320 dataset features (simplified)
| Variable | Description |
time(target) | Follow-up time to AIDS progression or death (in days). Represents the time from enrollment to the event (end of study or death). |
age | Age of the patient at the time of enrollment (in years). |
cd4_cell_count | Baseline CD4 T-cell count (cells/mL), a key indicator of immune function. |
race_* | Indicator variables representing the patient’s race. |
group_* | Indicator variables representing the treatment group. |
The associated dataset contains aprox. 1,150 records of HIV-infected patients who were randomized to receive either the novel triple-drug regimen or the conventional two-drug therapy.
73 Quantile regression
The target variable is time, representing the follow-up duration. Linear quantile regression
was used to estimate the impact of various linear predictors from Table 1 on the time to AIDS progression or death.
Quantile regression coefficients as functions of quantile are plotted in Figure 12. Low values represent individuals who progressed to AIDS or died quickly, while high values correspond to individuals with longer survival times.
Check (Koenker & Hallock, 2001) for more examples.
74 Baseline estimate
Baseline survival time is estimated by the model intercept (Figure 13), e.g., median intercept is approximately 240 days. Note that the intercept would be the median survival time if all other predictors were zero, in our case, they are not.
75 Reliability of coefficients
For , the estimates are most reliable and often close to the OLS estimates. Extreme quantiles are estimated at tails where data is sparse, leading to higher variance and less reliable estimates, as seen in the fluctuations in Figure 17 at both tails. Quantiles and were not estimated at all.
76 Sign of quantile regression coefficients
The sign of reflects the predictor ‘s impact on survival time at the -quantile, i.e., at the -quantile.
Consistently positive across all suggest that the predictor has only positive contributions to survival time for all individuals. For the indinavir group (Figure 14), the positive impact (in days) is greatest for short-survived patients (low ) and decreases for long-lived patients (high ).
Likewise, consistently negative across all suggest that the predictor has only negative contributions to survival time for all quantiles. AIDS patients generally have lower CD4 cell counts than healthy individuals, and the lower the CD4 cell count, the more pronounced its negative contribution (Figure 16) to survival time .
8 Goodness-of-fit
81 Bad metrics
Classical metrics (e.g., MAE, MSE, ) evaluate predictions based on their distribution around the mean . However, quantile regression focuses on other distribution properties, intentionally ignoring the mean. As a result, classical metrics are not suitable for evaluating quantile regression models.
In fact, MAE is equivalent to the mean quantile loss for , making it suitable for median regression specifically
For a model : For unbiased models, , which is used to derive (36).
82 Mean quantile loss
The simplest approach to evaluate quantile regression models is to use the quantile loss directly:
where is the quantile regression model.
For two quantile regression models and (e.g., for different quantiles, regularization, or features), the model with the lower quantile loss better fits the data and is preferred. In sklearn, this metric is implemented as sklearn.metrics.mean_pinball_loss.
83 metric
Another approach involves metrics specifically designed for quantile regression. Classical measures the proportion of variance explained by the model:
where is the sum of squared residuals between the predicted and actual values, and is the squared difference between the actual values and the mean . can be viewed as for a very simple constant model :
where and are the residual sum of squares for the actual (proposed) model and the mean constant (baseline) model , respectively.
For quantile regression, a similar metric can be defined ((Koenker & Machado, 1999), eq. 7). For two quantile regression models and and corresponding mean quantile losses and computed via (35), the analog of is:
where and are the proposed and baseline models, respectively. Usually, models are compared for the same quantile .
A straightforward choice for the baseline model is the empirical quantile value calculated from the training set. The difference in the upper index arises because in , quadratic units (, , and ) are used, while in , linear units (quantile loss) are used, as seen in (8).
Like the general definition of , which is not bound to and can be negative, the metric can also be negative if the model is worse than the baseline model . In sklearn, this metric is implemented as sklearn.metrics.d2_pinball_score.
84 Ordered metrics
A quantile model can be evaluated on how well it preserves the order. This is particularly important for risk modeling applications and ranking. For example, if a patient died at and another at , where , the model should predict .
In (39), the numerator counts the number of observation pairs where the model predicts the same order as the actual order. The denominator counts the total number of comparable observation pairs. Equation (39) can also be extended to handle censored data.
The calculation can be optimized by summing over unique pairs and , reducing redundancy.
The proportion of correctly ordered pairs is measured by the concordance index (C-index):
The C-index ranges from 0.5 (random predictions) to 1 (perfect predictions).
9 Practical considerations
91 Targets
Median is sometimes more interpretable and a better measure of centrality than the mean, particularly for skewed or multimodal data:
- Median salary or house price characterizes the central tendency of a distribution better than the mean, which can be skewed by extreme values.
92 Parameters
Coefficients in linear quantile regression are noisier than in OLS and depend on quantile , making them harder to interpret. The Gauss-Markov theorem ensuring convergence and variance in OLS does not apply to quantile regression.
Exact values of in OLS are interpretable, but in quantile regression, they are generally not. In simple cases, they can be close to OLS coefficients and interpretable. However, when quantile regression is applied to transformed data (e.g., ), coefficients remain invariant, but their contribution to becomes less obvious. For skewed data where OLS fails, quantile regression coefficients differ significantly from OLS but may still be interpretable.
93 Computational complexity
Quantile regression lacks a universal analytical solution and is typically solved numerically. The quantile loss function (8) combines two linear functions separated at . Residuals can be decomposed into positive and negative parts:
Using this decomposition, the quantile loss can be expressed as:
This formulation leads to a constrained linear programming problem [(Koenker et al., 2018), p.282]:
Solving this optimization problem is computationally more intensive than OLS’s closed-form solution, particularly for large datasets or when estimating multiple quantiles simultaneously.
94 Extreme quantiles
Estimates for extreme quantiles (e.g., or ) are often less reliable due to sparse data in distribution tails, resulting in higher variance as shown in the parameter convergence section.
95 Complete picture of conditional distributions
Quantile regression allows modeling multiple quantiles, providing a comprehensive view of how predictors affect the entire conditional distribution of the response, not just its center. This reveals heterogeneous effects that OLS cannot capture.
References
- Koenker, R. (2005). Quantile Regression (Repr, Issue 38). Cambridge Univ. Press.
- Koenker, R., & Bassett, G. (1978). Regression Quantiles. Econometrica, 46(1), 33. https://doi.org/10.2307/1913643
- Koenker, R., & Hallock, K. F. (2001). Quantile Regression. Journal of Economic Perspectives, 15(4), 143–156. https://doi.org/10.1257/jep.15.4.143
- Koenker, R., & Machado, J. A. F. (1999). Goodness of Fit and Related Inference Processes for Quantile Regression. Journal of the American Statistical Association, 94(448), 1296–1310. https://doi.org/10.1080/01621459.1999.10473882
- Koenker, R., Chernozhukov, V., He, X., & Peng, L. (2018). Handbook of Quantile Regression.