Conformal prediction

Conformal prediction (CP) is a machine learning framework for uncertainty quantification that produces statistically valid prediction regions (<a href="/facts/Prediction_interval/0twP0IFI">prediction intervals</a>) for any underlying point predictor (whether statistical, machine, or deep learning) only assuming <a href="/facts/Exchangeable_random_variables/tHvmhbRy">exchangeability</a> of the data. CP works by computing nonconformity scores on previously labeled data, and using these to create prediction sets on a new (unlabeled) test data point. A <a href="/facts/Transduction_(machine_learning)/YBMCKGfo">transductive</a> version of CP was first proposed in 1998 by <a href="/facts/Alexander_Gammerman/gXECIh6K">Gammerman</a>, <a href="/facts/Vladimir_Vovk/cKmlKfLV">Vovk</a>, and <a href="/facts/Vladimir_Vapnik/UQfpyUuN">Vapnik</a>, and since, several variants of conformal prediction have been developed with different computational complexities, formal guarantees, and practical applications.
Conformal prediction requires a user-specified significance level for which the algorithm should produce its predictions. This significance level restricts the frequency of errors that the algorithm is allowed to make. For example, a significance level of 0.1 means that the algorithm can make at most 10% erroneous predictions. To meet this requirement, the output is a set prediction, instead of a point prediction produced by standard <a href="/facts/Supervised_learning/JlNFNPFt">supervised machine learning</a> models. For classification tasks, this means that predictions are not a single class, for example 'cat', but instead a set like {'cat', 'dog'}. Depending on how good the underlying model is (how well it can discern between cats, dogs and other animals) and the specified significance level, these sets can be smaller or larger. For regression tasks, the output is prediction intervals, where a smaller significance level (fewer allowed errors) produces wider intervals which are less specific, and vice versa – more allowed errors produce tighter prediction intervals.

Conformal prediction open-in-new

Conformal prediction