Incremental decision tree

<h2 id="applications">Applications</h2>
<ul><li>On-line learning</li>
<li>Data streams</li>
<li><a href="/facts/Concept_drift/gLQmz1Rc">Concept drift</a></li>
<li>Data which can be modeled well using a hierarchical model.</li>
<li>Systems where a user-interpretable output is desired.</li></ul>
<h2 id="methods">Methods</h2>
Here is a short list of incremental decision tree methods, organized by their (usually non-incremental) parent algorithms.

<h3>CART family</h3>
<a href="/facts/Predictive_analytics/tbqzk5qe">CART</a><a class="footnote-ref" id="fnref:1" href="#fn:1">1</a> (1984) is a nonincremental decision tree inducer for both classification and regression problems. developed in the mathematics and statistics communities. CART traces its roots to AID (1963)<a class="footnote-ref" id="fnref:2" href="#fn:2">2</a> 

<ul><li>incremental CART (1989)<a class="footnote-ref" id="fnref:3" href="#fn:3">3</a> Crawford modified CART to incorporate data incrementally.</li></ul>
<h3>ID3/C4.5 family</h3>
<a href="/facts/ID3_algorithm/cey9VCT7">ID3</a> (1986)<a class="footnote-ref" id="fnref:4" href="#fn:4">4</a> and <a href="/facts/C4.5/GGsxBTtz">C4.5</a> (1993)<a class="footnote-ref" id="fnref:5" href="#fn:5">5</a> were developed by <a href="/facts/Ross_Quinlan/skDLL2u5">Quinlan</a> and have roots in Hunt's Concept Learning System (CLS, 1966)<a class="footnote-ref" id="fnref:6" href="#fn:6">6</a> The ID3 family of tree inducers was developed in the engineering and computer science communities.

<ul><li>ID3' (1986)<a class="footnote-ref" id="fnref:7" href="#fn:7">7</a> was suggested by Schlimmer and Fisher. It was a brute-force method to make ID3 incremental; after each new data instance is acquired, an entirely new tree is induced using ID3.</li>
<li>ID4 (1986)<a class="footnote-ref" id="fnref:8" href="#fn:8">8</a> could incorporate data incrementally. However, certain concepts were unlearnable, because ID4 discards subtrees when a new test is chosen for a node.</li>
<li>ID5 (1988)<a class="footnote-ref" id="fnref:9" href="#fn:9">9</a> didn't discard subtrees, but also did not guarantee that it would produce the same tree as ID3.</li>
<li>ID5R (1989)<a class="footnote-ref" id="fnref:10" href="#fn:10">10</a> output the same tree as ID3 for a dataset regardless of the incremental training order. This was accomplished by recursively updating the tree's subnodes. It did not handle numeric variables, <a href="/facts/Multiclass_classification/aS7sV9EX">multiclass classification</a> tasks, or missing values.</li>
<li>ID6MDL (2007)<a class="footnote-ref" id="fnref:11" href="#fn:11">11</a> an extended version of the ID3 or ID5R algorithms.</li>
<li>ITI (1997)<a class="footnote-ref" id="fnref:12" href="#fn:12">12</a> is an efficient method for incrementally inducing decision trees. The same tree is produced for a dataset regardless of the data's presentation order, or whether the tree is induced incrementally or non incrementally (batch mode). It can accommodate numeric variables, multiclass tasks, and missing values. Code is available on the web. <a href="http://www-lrn.cs.umass.edu/iti/index.html">[1]</a></li></ul>
note: ID6NB (2009)<a class="footnote-ref" id="fnref:13" href="#fn:13">13</a> is not incremental.

<h3>Other Incremental Learning Systems</h3>
There were several incremental concept learning systems that did not build decision trees, but which predated and influenced the development of the earliest incremental decision tree learners, notably ID4.<a class="footnote-ref" id="fnref:14" href="#fn:14">14</a> Notable among these was Schlimmer and Granger's STAGGER (1986),<a class="footnote-ref" id="fnref:15" href="#fn:15">15</a> which learned disjunctive concepts incrementally. STAGGER was developed to examine concepts that changed over time (<a href="/facts/Concept_drift/gLQmz1Rc">concept drift</a>). Prior to STAGGER, Michalski and Larson (1978)<a class="footnote-ref" id="fnref:16" href="#fn:16">16</a> investigated an incremental variant of AQ (Michalski, 1973),<a class="footnote-ref" id="fnref:17" href="#fn:17">17</a> a supervised system for learning concepts in <a href="/facts/Disjunctive_normal_form/hsw2pRPu">disjunctive normal form</a> (DNF). Experience with these earlier systems and others, to include incremental tree-structured unsupervised learning, contributed to a conceptual framework for evaluating incremental decision tree learners specifically, and incremental concept learning generally, along four dimensions that reflect the inherent tradeoffs between learning cost and quality:<a class="footnote-ref" id="fnref:18" href="#fn:18">18</a> (1) cost of knowledge base update, (2) the number of observations that are required to converge on a knowledge base with given characteristics, (3) the total effort (as a function of the first two dimensions) that a system exerts, and the (4) quality (often consistency) of the final knowledge base. Some of the historical context in which incremental decision tree learners emerged is given in Fisher and Schlimmer (1988),<a class="footnote-ref" id="fnref:19" href="#fn:19">19</a> and which also expands on the four factor framework that was used to evaluate and design <a href="/facts/Incremental_learning/MQ3cElgK">incremental learning</a> systems.

<h3>VFDT Algorithm</h3>
Very Fast Decision Trees learner reduces training time for large incremental data sets by subsampling the incoming data stream.

<ul><li>VFDT (2000)<a class="footnote-ref" id="fnref:20" href="#fn:20">20</a></li>
<li>CVFDT (2001)<a class="footnote-ref" id="fnref:21" href="#fn:21">21</a> can adapt to <a href="/facts/Concept_drift/gLQmz1Rc">concept drift</a>, by using a sliding window on incoming data. Old data outside the window is forgotten.</li>
<li>VFDTc (2006)<a class="footnote-ref" id="fnref:22" href="#fn:22">22</a> extends VFDT for continuous data, concept drift, and application of Naive Bayes classifiers in the leaves.</li>
<li>VFML (2003) is a toolkit and available on the web. <a href="http://www.cs.washington.edu/dm/vfml/">[2]</a>. It was developed by the creators of VFDT and CVFDT.</li></ul>
<h3>OLIN and IFN</h3>
<ul><li>OLIN (2002)<a class="footnote-ref" id="fnref:23" href="#fn:23">23</a></li>
<li>IOLIN (2008)<a class="footnote-ref" id="fnref:24" href="#fn:24">24</a> — based on Info-Fuzzy Network (IFN)<a class="footnote-ref" id="fnref:25" href="#fn:25">25</a></li></ul>
<h3>GAENARI</h3>
<ul><li><a href="https://github.com/greenfish77/gaenari">gaenari</a></li></ul>
<h2 id="see-also">See also</h2>
<ul><li><a href="/facts/Concept_drift/gLQmz1Rc">Concept drift</a></li>
<li><a href="/facts/Decision_tree_learning/LbPMAwdI">Decision tree</a></li>
<li><a href="/facts/Machine_Learning/e0w0XJTu">Machine Learning</a></li>
<li><a href="/facts/Online_machine_learning/4BoJlDfg">Online learning</a></li></ul>

<h2 id="external-links">External links</h2>
<ul><li>ITI code. <a href="http://www-lrn.cs.umass.edu/iti/index.html">http://www-lrn.cs.umass.edu/iti/index.html</a></li>
<li>VFML code. <a href="http://www.cs.washington.edu/dm/vfml/">http://www.cs.washington.edu/dm/vfml/</a></li>
<li>C++ incremental decision tree. <a href="https://github.com/greenfish77/gaenari">https://github.com/greenfish77/gaenari</a></li></ul>

<h2 id="references">References</h2>

<ol>
<li id="fn:1">Breiman, L.; Friedman, J.H.; Olshen, R.A.; Stone, C.J. (1984). Classification and regression trees. Belmont, CA: Wadsworth International. ISBN 978-1-351-46048-4. <a href="978-1-351-46048-4" target="_blank">978-1-351-46048-4</a> <a href="#fnref:1" class="footnote-back-ref">↩</a></li>
<li id="fn:2">Morgan, J.N; Sondquist, J.A. (1963). "Problems in the analysis of survey data, and a proposal" (PDF). J. Amer. Statist. Assoc. 58 (302): 415–434. doi:10.1080/01621459.1963.10500855. <a href="http://cda.psych.uiuc.edu/statistical_learning_course/morgan_sonquist.pdf" target="_blank">http://cda.psych.uiuc.edu/statistical_learning_course/morgan_sonquist.pdf</a> <a href="#fnref:2" class="footnote-back-ref">↩</a></li>
<li id="fn:3">Crawford, S.L. (1989). "Extensions to the CART algorithm". International Journal of Man-Machine Studies. 31 (2): 197–217. doi:10.1016/0020-7373(89)90027-8. <a href="https://dx.doi.org/10.1016/0020-7373%2889%2990027-8" target="_blank">https://dx.doi.org/10.1016/0020-7373%2889%2990027-8</a> <a href="#fnref:3" class="footnote-back-ref">↩</a></li>
<li id="fn:4">Quinlan, J.R. (1986). "Induction of Decision Trees" (PDF). Machine Learning. 1 (1): 81–106. doi:10.1007/BF00116251. S2CID 13252401. <a href="https://link.springer.com/content/pdf/10.1007/BF00116251.pdf" target="_blank">https://link.springer.com/content/pdf/10.1007/BF00116251.pdf</a> <a href="#fnref:4" class="footnote-back-ref">↩</a></li>
<li id="fn:5">Quinlan, J.R. (2014) [1993]. C4.5: Programs for machine learning. Elsevier. ISBN 978-1-55860-238-0. <a href="978-1-55860-238-0" target="_blank">978-1-55860-238-0</a> <a href="#fnref:5" class="footnote-back-ref">↩</a></li>
<li id="fn:6">Hunt, E.B.; Marin, J.; Stone, P.J. (1966). Experiments in induction. Academic Press. ISBN 978-0-12-362350-8. <a href="978-0-12-362350-8" target="_blank">978-0-12-362350-8</a> <a href="#fnref:6" class="footnote-back-ref">↩</a></li>
<li id="fn:7">Schlimmer, J.C.; Fisher, D. (1986). "A case study of incremental concept induction". AAAI'86: Proceedings of the Fifth National Conference on Artificial Intelligence. Morgan Kaufmann. pp. 496–501. <a href="https://www.researchgate.net/publication/221603307" target="_blank">https://www.researchgate.net/publication/221603307</a> <a href="#fnref:7" class="footnote-back-ref">↩</a></li>
<li id="fn:8">Schlimmer, J.C.; Fisher, D. (1986). "A case study of incremental concept induction". AAAI'86: Proceedings of the Fifth National Conference on Artificial Intelligence. Morgan Kaufmann. pp. 496–501. <a href="https://www.researchgate.net/publication/221603307" target="_blank">https://www.researchgate.net/publication/221603307</a> <a href="#fnref:8" class="footnote-back-ref">↩</a></li>
<li id="fn:9">Utgoff, P.E. (1988). "ID5: An incremental ID3" (PDF). Machine Learning Proceedings 1988 Proceedings of the Fifth International Conference on Machine Learning. Morgan Kaufmann. pp. 107–120. doi:10.1016/B978-0-934613-64-4.50017-7. ISBN 978-0-934613-64-4.
Publishers. <a href="978-0-934613-64-4" target="_blank">978-0-934613-64-4</a> <a href="#fnref:9" class="footnote-back-ref">↩</a></li>
<li id="fn:10">Utgoff, P.E. (1989). "Incremental induction of decision trees" (PDF). Machine Learning. 4 (2): 161–186. doi:10.1023/A:1022699900025. S2CID 5293072. <a href="https://link.springer.com/content/pdf/10.1023/A:1022699900025.pdf" target="_blank">https://link.springer.com/content/pdf/10.1023/A:1022699900025.pdf</a> <a href="#fnref:10" class="footnote-back-ref">↩</a></li>
<li id="fn:11">Kroon, M., Korzec, S., Adriani, P. (2007) ID6MDL: Post-Pruning Incremental Decision Trees. <a href="#fnref:11" class="footnote-back-ref">↩</a></li>
<li id="fn:12">Utgoff, P.E.; Berkman, N.C.; Clouse, J.A. (1997). "Decision tree induction based on efficient tree restructuring" (PDF). Machine Learning. 29: 5–44. doi:10.1023/A:1007413323501. S2CID 2743403. <a href="https://link.springer.com/content/pdf/10.1023/A:1007413323501.pdf" target="_blank">https://link.springer.com/content/pdf/10.1023/A:1007413323501.pdf</a> <a href="#fnref:12" class="footnote-back-ref">↩</a></li>
<li id="fn:13">Appavu, S.; Rajaram, R. (2009). "Knowledge-based system for text classification using ID6NB algorithm]". Knowledge-Based Systems. 22 (1): 1–7. doi:10.1016/j.knosys.2008.04.006. <a href="https://www.sciencedirect.com/science/article/pii/S0950705108001081" target="_blank">https://www.sciencedirect.com/science/article/pii/S0950705108001081</a> <a href="#fnref:13" class="footnote-back-ref">↩</a></li>
<li id="fn:14">Schlimmer, J.C.; Fisher, D. (1986). "A case study of incremental concept induction". AAAI'86: Proceedings of the Fifth National Conference on Artificial Intelligence. Morgan Kaufmann. pp. 496–501. <a href="https://www.researchgate.net/publication/221603307" target="_blank">https://www.researchgate.net/publication/221603307</a> <a href="#fnref:14" class="footnote-back-ref">↩</a></li>
<li id="fn:15">Schlimmer, J.C.; Granger, Jr., R.H. (1986). "Incremental learning from noisy data". Machine Learning. 1 (3): 317–354. doi:10.1007/BF00116895. S2CID 33776987. <a href="https://doi.org/10.1007%2FBF00116895" target="_blank">https://doi.org/10.1007%2FBF00116895</a> <a href="#fnref:15" class="footnote-back-ref">↩</a></li>
<li id="fn:16">Michalski, R.S.; Larson, J.B. (1978). Selection of most representative training examples and incremental generation of VL hypotheses: The underlying methodology and the description of the programs ESEL and AQ11 (PDF) (Technical report). University of Illinois, Department of Computer Science. hdl:1920/1544/78-03. UIUCDCS-R-78-867. <a href="http://digilib.gmu.edu/jspui/bitstream/handle/1920/1544/78-03.pdf" target="_blank">http://digilib.gmu.edu/jspui/bitstream/handle/1920/1544/78-03.pdf</a> <a href="#fnref:16" class="footnote-back-ref">↩</a></li>
<li id="fn:17">Michalski, R.S. (1973). "Discovering classification rules using variable-valued logic system VL1" (PDF). IJCAI'73: Proceedings of the Third International Joint Conference on Artificial Intelligence. Stanford, CA: Morgan Kaufmann. pp. 162–172. hdl:1920/1515/73-01. <a href="http://digilib.gmu.edu/jspui/bitstream/handle/1920/1515/73-01.pdf" target="_blank">http://digilib.gmu.edu/jspui/bitstream/handle/1920/1515/73-01.pdf</a> <a href="#fnref:17" class="footnote-back-ref">↩</a></li>
<li id="fn:18">Schlimmer, J.C.; Fisher, D. (1986). "A case study of incremental concept induction". AAAI'86: Proceedings of the Fifth National Conference on Artificial Intelligence. Morgan Kaufmann. pp. 496–501. <a href="https://www.researchgate.net/publication/221603307" target="_blank">https://www.researchgate.net/publication/221603307</a> <a href="#fnref:18" class="footnote-back-ref">↩</a></li>
<li id="fn:19">Fisher, D.; Schlimmer, J. (1988). Models of Incremental Concept Learning: A coupled research proposal (Technical report). Vanderbilt University. CS-88-05. <a href="http://www.vuse.vanderbilt.edu/~dfisher/tech-reports/tr-88-05/proposal.html" target="_blank">http://www.vuse.vanderbilt.edu/~dfisher/tech-reports/tr-88-05/proposal.html</a> <a href="#fnref:19" class="footnote-back-ref">↩</a></li>
<li id="fn:20">Domingos, P.; Hulten, G. (2000). "Mining high-speed data streams" (PDF). Proceedings KDD Proceedings of the sixth ACM SIGKDD international conference on Knowledge discovery and data mining. ACM Press. pp. 71–80. doi:10.1145/347090.347107. ISBN 1-58113-233-6. S2CID 8810610. <a href="1-58113-233-6" target="_blank">1-58113-233-6</a> <a href="#fnref:20" class="footnote-back-ref">↩</a></li>
<li id="fn:21">Hulten, G.; Spencer, L.; Domingos, P. (2001). "Mining time-changing data streams" (PDF). Proceedings of the seventh ACM SIGKDD international conference on Knowledge discovery and data mining. ACM Press. pp. 97–106. doi:10.1145/502512.502529. ISBN 978-1-58113-391-2. S2CID 6416602. <a href="978-1-58113-391-2" target="_blank">978-1-58113-391-2</a> <a href="#fnref:21" class="footnote-back-ref">↩</a></li>
<li id="fn:22">Gama, J.; Fernandes, R.; Rocha, R. (2006). "Decision trees for mining data streams". Intelligent Data Analysis. 10: 23–45. doi:10.3233/IDA-2006-10103. <a href="https://content.iospress.com/articles/intelligent-data-analysis/ida00234" target="_blank">https://content.iospress.com/articles/intelligent-data-analysis/ida00234</a> <a href="#fnref:22" class="footnote-back-ref">↩</a></li>
<li id="fn:23">Last, M. (2002). "Online classification of nonstationary data streams" (PDF). Intell. Data Anal. 6 (2): 129–147. doi:10.3233/IDA-2002-6203. <a href="https://www.academia.edu/download/46518868/Change2.pdf" target="_blank">https://www.academia.edu/download/46518868/Change2.pdf</a> <a href="#fnref:23" class="footnote-back-ref">↩</a></li>
<li id="fn:24">Cohen, L.; Avrahami, G.; Last, M.; Kandel, A. (2008). "Info-fuzzy algorithms for mining dynamic data streams" (PDF). Applied Soft Computing. 8 (4): 1283–94. doi:10.1016/j.asoc.2007.11.003. <a href="http://www.ise.bgu.ac.il/faculty/mlast/papers/Paper_RTDM_ASOC_SI_Final.pdf" target="_blank">http://www.ise.bgu.ac.il/faculty/mlast/papers/Paper_RTDM_ASOC_SI_Final.pdf</a> <a href="#fnref:24" class="footnote-back-ref">↩</a></li>
<li id="fn:25">Maimon, O.; Last, M. (2000). The info-fuzzy network (IFN) methodology. Knowledge Discovery and Data Mining. Kluwer. doi:10.1007/978-1-4757-3296-2. ISBN 978-1-4757-3296-2. S2CID 41520652. <a href="978-1-4757-3296-2" target="_blank">978-1-4757-3296-2</a> <a href="#fnref:25" class="footnote-back-ref">↩</a></li>
</ol>

Incremental decision tree open-in-new

Incremental decision tree