Inductive bias

<h2 id="types">Types</h2>
<p>The following is a list of common inductive biases in machine learning algorithms.
</p>
<ul><li>Maximum <a href="/facts/Conditional_independence/udUXFaMw">conditional independence</a>: if the hypothesis can be cast in a <a href="/facts/Bayesian_inference/tEpK3zLx">Bayesian</a> framework, try to maximize conditional independence. This is the bias used in the <a href="/facts/Naive_Bayes_classifier/FmNHI688">Naive Bayes classifier</a>.</li>
<li>Minimum <a href="/facts/Cross-validation_(statistics)/pDWg8s0e">cross-validation</a> error: when trying to choose among hypotheses, select the hypothesis with the lowest cross-validation error. Although cross-validation may seem to be free of bias, the <a href="/facts/No_free_lunch_theorem/JoSR8TTC">"no free lunch"</a> theorems show that cross-validation must be biased, for example assuming that there is no information encoded in the ordering of the data.</li>
<li>Maximum margin: when drawing a boundary between two classes, attempt to maximize the width of the boundary. This is the bias used in <a href="/facts/Support_vector_machines/XobxpdBG">support vector machines</a>. The assumption is that distinct classes tend to be separated by wide boundaries.</li>
<li><a href="/facts/Minimum_description_length/em8tXqjf">Minimum description length</a>: when forming a hypothesis, attempt to minimize the length of the description of the hypothesis.</li>
<li>Minimum features: unless there is good evidence that a <a href="/facts/Feature_space/nzAhYxfu">feature</a> is useful, it should be deleted. This is the assumption behind <a href="/facts/Feature_selection/gxbuF4VG">feature selection</a> algorithms.</li>
<li>Nearest neighbors: assume that most of the cases in a small neighborhood in <a href="/facts/Feature_space/nzAhYxfu">feature space</a> belong to the same class. Given a case for which the class is unknown, guess that it belongs to the same class as the majority in its immediate neighborhood. This is the bias used in the <a href="/facts/K-nearest_neighbors_algorithm/Bs1UaAmE">k-nearest neighbors algorithm</a>. The assumption is that cases that are near each other tend to belong to the same class.</li></ul>
<h2 id="shift-of-bias">Shift of bias</h2>
<p>Although most learning algorithms have a static bias, some algorithms are designed to shift their bias as they acquire more data.<a class="footnote-ref" id="fnref:6" href="#fn:6"><sup>6</sup></a> This does not avoid bias, since the bias shifting process itself must have a bias.
</p>
<h2 id="see-also">See also</h2>
<ul><li><a href="/facts/Algorithmic_bias/vHaHsIdV">Algorithmic bias</a></li>
<li><a href="/facts/Cognitive_bias/49YKVhmN">Cognitive bias</a></li>
<li><a href="/facts/No_free_lunch_theorem/JoSR8TTC">No free lunch theorem</a></li>
<li><a href="/facts/No_free_lunch_in_search_and_optimization/aNGHqkID">No free lunch in search and optimization</a></li></ul>

<h2 id="references">References</h2>

<ol>
<li id="fn:1"><p>Mitchell, T. M. (1980), The need for biases in learning generalizations, CBM-TR 5-110, New Brunswick, New Jersey, USA: Rutgers University, CiteSeerX 10.1.1.19.5466 <a href="/wiki/CiteSeerX_(identifier)" target="_blank">/wiki/CiteSeerX_(identifier)</a> <a href="#fnref:1" class="footnote-back-ref">↩</a></p></li>
<li id="fn:2"><p>Goodman, Nelson (1955). "The new riddle of induction". Fact, Fiction, and Forecast. Harvard University Press. pp. 59–83. ISBN 978-0-674-29071-6. {{cite book}}: ISBN / Date incompatibility (help) <a href="978-0-674-29071-6" target="_blank">978-0-674-29071-6</a> <a href="#fnref:2" class="footnote-back-ref">↩</a></p></li>
<li id="fn:3"><p>Mitchell, Tom M (1980). "The need for biases in learning generalizations" (PDF). Rutgers University Technical Report CBM-TR-117: 184–191. <a href="https://axon.cs.byu.edu/~martinez/classes/678/Papers/Mitchell_IB.pdf" target="_blank">https://axon.cs.byu.edu/~martinez/classes/678/Papers/Mitchell_IB.pdf</a> <a href="#fnref:3" class="footnote-back-ref">↩</a></p></li>
<li id="fn:4"><p>Mitchell, T. M. (1980), The need for biases in learning generalizations, CBM-TR 5-110, New Brunswick, New Jersey, USA: Rutgers University, CiteSeerX 10.1.1.19.5466 <a href="/wiki/CiteSeerX_(identifier)" target="_blank">/wiki/CiteSeerX_(identifier)</a> <a href="#fnref:4" class="footnote-back-ref">↩</a></p></li>
<li id="fn:5"><p>DesJardins, M.; Gordon, D. F. (1995), "Evaluation and selection of biases in machine learning", Machine Learning, 20 (1–2): 5–22, doi:10.1007/BF00993472 <a href="https://link.springer.com/article/10.1007/BF00993472" target="_blank">https://link.springer.com/article/10.1007/BF00993472</a> <a href="#fnref:5" class="footnote-back-ref">↩</a></p></li>
<li id="fn:6"><p>Utgoff, P. E. (1984), Shift of bias for inductive concept learning, New Brunswick, New Jersey, USA: Doctoral dissertation, Department of Computer Science, Rutgers University, ISBN 9780934613002 <a href="9780934613002" target="_blank">9780934613002</a> <a href="#fnref:6" class="footnote-back-ref">↩</a></p></li>
</ol>

Inductive bias open-in-new

Inductive bias