Testing hypotheses suggested by the data

<h2 id="the-general-problem">The general problem</h2>
<p>Testing a hypothesis suggested by the data can very easily result in false positives (<a href="/facts/Type_I_error/aazbF3fq">type I errors</a>). If one looks long enough and in enough different places, eventually data can be found to support any hypothesis. Yet, these positive data do not by themselves constitute <a href="/facts/Scientific_evidence/ChQVqPcP">evidence</a> that the hypothesis is correct. The negative test data that were thrown out are just as important, because they give one an idea of how common the positive results are compared to chance. Running an experiment, seeing a pattern in the data, proposing a hypothesis from that pattern, then using the <i>same</i> experimental data as evidence for the new hypothesis is extremely suspect, because data from all other experiments, completed or potential, has essentially been "thrown out" by choosing to look only at the experiments that suggested the new hypothesis in the first place.
</p><p>A large set of tests as described above greatly inflates the <a href="/facts/Probability/fgYvMhwb">probability</a> of <a href="/facts/Type_I_error/aazbF3fq">type I error</a> as all but the data most favorable to the <a href="/facts/Hypothesis/gMABEJul">hypothesis</a> is discarded. This is a risk, not only in <a href="/facts/Statistical_hypothesis_testing/yv9RPF6U">hypothesis testing</a> but in all <a href="/facts/Statistical_inference/sJLjubxm">statistical inference</a> as it is often problematic to accurately describe the process that has been followed in searching and discarding <a href="/facts/Data/AUBFN97o">data</a>. In other words, one wants to keep all data (regardless of whether they tend to support or refute the hypothesis) from "good tests", but it is sometimes difficult to figure out what a "good test" is. It is a particular problem in <a href="/facts/Statistical_model/jKkT5ftm">statistical modelling</a>, where many different models are rejected by <a href="/facts/Trial_and_error/NGm2GoEd">trial and error</a> before publishing a result (see also <a href="/facts/Overfitting/5xnFLcMg">overfitting</a>, <a href="/facts/Publication_bias/Wbmr1WjW">publication bias</a>).
</p><p>The error is particularly prevalent in <a href="/facts/Data_mining/AH53q5ac">data mining</a> and <a href="/facts/Machine_learning/e0w0XJTu">machine learning</a>. It also commonly occurs in <a href="/facts/Academic_publishing/WfPqsYr1">academic publishing</a> where only reports of positive, rather than negative, results tend to be accepted, resulting in the effect known as <a href="/facts/Publication_bias/Wbmr1WjW">publication bias</a>.
</p>
<h2 id="correct-procedures">Correct procedures</h2>
<p>All strategies for sound testing of hypotheses suggested by the data involve including a wider range of tests in an attempt to validate or refute the new hypothesis. These include:
</p>
<ul><li>Collecting confirmation samples</li>
<li><a href="/facts/Cross-validation_(statistics)/pDWg8s0e">Cross-validation</a></li>
<li>Methods of compensation for <a href="/facts/Multiple_comparisons/lAItaL18">multiple comparisons</a></li>
<li>Simulation studies including adequate representation of the multiple-testing actually involved</li></ul>
<p><a href="/facts/Scheff%25C3%25A9_test/Qxt0sIJo">Henry Scheffé's simultaneous test</a> of all contrasts in <a href="/facts/Multiple_comparisons/lAItaL18">multiple comparison</a> problems is the most well-known remedy in the case of <a href="/facts/Analysis_of_variance/HHR2jkDk">analysis of variance</a>.<a class="footnote-ref" id="fnref:1" href="#fn:1"><sup>1</sup></a> It is a method designed for testing hypotheses suggested by the data while avoiding the fallacy described above.
</p>
<h2 id="see-also">See also</h2>
<ul><li><a href="/facts/Bonferroni_correction/br48S6qq">Bonferroni correction</a></li>
<li><a href="/facts/Data_analysis/CNtARxLW">Data analysis</a></li>
<li><a href="/facts/Data_dredging/X11G2jBn">Data dredging</a>, <i>p</i>-hacking</li>
<li><a href="/facts/Exploratory_data_analysis/8hvnf7JS">Exploratory data analysis</a></li>
<li><a href="/facts/HARKing/IshjE8cn">HARKing</a></li>
<li><a href="/facts/Post_hoc_analysis/mCNhrGls">Post hoc analysis</a></li>
<li><a href="/facts/Predictive_analytics/tbqzk5qe">Predictive analytics</a></li>
<li><a href="/facts/Texas_sharpshooter_fallacy/9xvaNM4Q">Texas sharpshooter fallacy</a></li>
<li><a href="/facts/Type_I_and_type_II_errors/aazbF3fq">Type I and type II errors</a></li>
<li><a href="/facts/Uncomfortable_science/HPwnb9Cb">Uncomfortable science</a></li></ul>
<h2 id="notes-and-references">Notes and references</h2>

<h2 id="references">References</h2>

<ol>
<li id="fn:1"><p>Henry Scheffé, "A Method for Judging All Contrasts in the Analysis of Variance", Biometrika, 40, pages 87–104 (1953). doi:10.1093/biomet/40.1-2.87 <a href="/wiki/Henry_Scheff%C3%A9" target="_blank">/wiki/Henry_Scheff%C3%A9</a> <a href="#fnref:1" class="footnote-back-ref">↩</a></p></li>
</ol>

Testing hypotheses suggested by the data open-in-new

Testing hypotheses suggested by the data