Iteratee

<h2 id="commonly-associated-abstractions">Commonly associated abstractions</h2>
<p>The following abstractions are not strictly speaking necessary to work with iteratees, but they do make it more convenient.
</p>
<h3>Enumerators</h3>
<p>An Enumerator is a convenient abstraction for feeding data into an iteratee from an arbitrary data source. Typically the enumerator will take care of any necessary resource cleanup associated with the data source. Because the enumerator knows exactly when the iteratee has finished reading data, it will do the resource cleanup (such as closing a file) at exactly the right time – neither too early nor too late. However, it can do this without needing to know about, or being co-located to, the implementation of the iteratee – so enumerators and iteratees form an example of <a href="/facts/Separation_of_concerns/Tx8jawNT">separation of concerns</a>.
</p>
<h3>Enumeratees</h3>
<p>An Enumeratee is a convenient abstraction for transforming the output of either an enumerator or iteratee, and feeding that output to an iteratee. For example, a "map" enumeratee would <a href="/facts/Map_(higher-order_function)/GdzN4UJl">map</a> a function over each input chunk.<a class="footnote-ref" id="fnref:4" href="#fn:4"><sup>4</sup></a>
</p>
<h2 id="motivations">Motivations</h2>
<p>Iteratees were created due to problems with existing <a href="/facts/Purely_functional_programming_language/am3DcNAd">purely functional</a> solutions to the problem of making <a href="/facts/Input%2foutput/WQTE8Jas">input/output</a> composable yet correct. Lazy I/O in Haskell allowed pure functions to operate on data on disk as if it were in memory, without explicitly doing I/O at all after opening the file - a kind of <a href="/facts/Memory-mapped_file/veYrNQnl">memory-mapped file</a> feature - but because it was impossible in general (due to the <a href="/facts/Halting_problem/Xz1bac40">Halting problem</a>) for the runtime to know whether the file or other resource was still needed, excessive numbers of files could be left open unnecessarily, resulting in <a href="/facts/File_descriptor/Iw74UKqG">file descriptor</a> exhaustion at the <a href="/facts/Operating_system/8XTuvMh2">operating system</a> level. Traditional <a href="/facts/C_(programming_language)/Ky2No763">C</a>-style I/O, on the other hand, was too low-level and required the developer to be concerned with low-level details such as the current position in the file, which hindered composability. Iteratees and enumerators combine the high-level functional programming benefits of lazy I/O, with the ability to control resources and low-level details where necessary afforded by C-style I/O.<a class="footnote-ref" id="fnref:5" href="#fn:5"><sup>5</sup></a>
</p>
<h2 id="examples">Examples</h2>

<h2 id="uses">Uses</h2>
<p>Iteratees are used in the Play framework to push data out to long-running <a href="/facts/Comet_(programming)/kLGmy68t">Comet</a> and <a href="/facts/WebSocket/NcQrqpnv">WebSocket</a> connections to <a href="/facts/Web_browsers/2pfyymLs">web browsers</a>.
</p><p>Iteratees may also be used to perform incremental <a href="/facts/Parsing/nRKB01S3">parsing</a> (that is, parsing that does not read all the data into memory at once), for example of <a href="/facts/JSON/bs1rUAhd">JSON</a>.<a class="footnote-ref" id="fnref:6" href="#fn:6"><sup>6</sup></a>
</p><p>Iteratees are a very general abstraction and can be used for arbitrary kinds of <a href="/facts/Sequential_access/8NCL847H">sequential</a> information processing (or mixed sequential/random-access processing) - and need not involve any I/O at all. This makes it easy to repurpose an iteratee to work on an in-memory dataset instead of data flowing in from the network.
</p>
<h2 id="history">History</h2>
<p>In a sense, a distant predecessor of the notion of an enumerator pushing data into a chain of one or more iteratees, was the <a href="/facts/Pipeline_(Unix)/EIhovuT2">pipeline</a> concept in operating systems. However, unlike a typical pipeline, iteratees are not separate processes (and hence do not have the overhead of <a href="/facts/Interprocess_communication/cIQYyT6y">IPC</a>) - or even separate threads, although they can perform work in a similar manner to a chain of worker threads sending messages to each other. This means that iteratees are more lightweight than processes or threads - unlike the situations with separate processes or threads, no extra stacks are needed.
</p><p>Iteratees and enumerators were invented by Oleg Kiselyov for use in Haskell.<a class="footnote-ref" id="fnref:7" href="#fn:7"><sup>7</sup></a> Later, they were introduced into Scalaz (in version 5.0; enumeratees were absent and were introduced in Scalaz 7) and into Play Framework 2.0.
</p>
<h2 id="formal-semantics">Formal semantics</h2>
<p>Iteratees have been formally modelled as <a href="/facts/Free_monad/kYazRoF1">free monads</a>, allowing equational laws to be validated, and employed to optimise programs using iteratees.<a class="footnote-ref" id="fnref:8" href="#fn:8"><sup>8</sup></a>
</p>
<h2 id="alternatives">Alternatives</h2>
<ul><li>Iterators may be used instead of iteratees in Scala, but they are <a href="/facts/Imperative_programming/6b5dUnE1">imperative</a>, so are not a <a href="/facts/Purely_functional_programming/am3DcNAd">purely functional</a> solution.</li>
<li>In Haskell, two alternative abstractions known as Conduits and Pipes have been developed. (These Pipes are not operating system level pipes, so like iteratees they do not require the use of <a href="/facts/System_calls/NW44AFJr">system calls</a>). Conduits in particular are associated with substantially richer libraries of primitives and combinators than iteratees; conduit adapters for incremental functionalities such as parsing HTML, XML, generalised parsing, making HTTP requests and processing the responses, exist, making conduits more suitable than iteratees for industrial software development in Haskell, out of the box.</li>
<li>There is also a high-level abstraction named <a href="http://hackage.haskell.org/package/machines">machines</a>. In Scala there is a package called <a href="https://github.com/functional-streams-for-scala/fs2">FS2: Functional Streams for Scala</a>, whose ancestry can be traced back to machines via several ports, renames and refactors.</li>
<li>In Haskell, the package <a href="http://hackage.haskell.org/cgi-bin/hackage-scripts/package/safe-lazy-io">safe-lazy-io</a> exists. It provides a simpler solution to some of the same problems, which essentially involves being "strict enough" to pull all data that is required, or might be required, through a pipeline which takes care of cleaning up the resources on completion.</li></ul>

<h2 id="further-reading">Further reading</h2>
<ul><li>John W. Lato (12 May 2010). <a href="http://themonadreader.wordpress.com/2010/05/12/issue-16/">"Iteratee: Teaching an Old Fold New Tricks"</a>. <i>Issue #16 of The Monad Reader</i>. Retrieved 29 June 2013. This relates to Haskell.</li></ul>
<h2 id="external-links">External links</h2>
<ul><li>Scala tutorials
<ul><li>Play 2.0
<ul><li><a href="http://mandubian.com/2012/08/27/understanding-play2-iteratees-for-normal-humans/">Understanding Play 2 iteratees for normal humans</a></li>
<li><a href="https://jazzy.id.au/default/2012/11/06/iteratees_for_imperative_programmers.html">Iteratees for imperative programmers</a></li></ul></li>
<li>Scalaz
<ul><li><a href="http://blog.higher-order.com/blog/2010/10/14/scalaz-tutorial-enumeration-based-io-with-iteratees/">Scalaz tutorial: Enumeration-based I/O with iteratees</a></li></ul></li></ul></li>
<li>Haskell tutorials
<ul><li><a href="http://www.scs.stanford.edu/11au-cs240h/notes/iteratee.html">Stanford lecture notes</a></li></ul></li>
<li>Further information
<ul><li><a href="http://okmij.org/ftp/Streams.html">Oleg Kiselyov's Iteratees and Enumerators page</a></li></ul></li></ul>

<h2 id="references">References</h2>

<ol>
<li id="fn:1"><p>"Handling data streams reactively". Play Framework documentation. Retrieved 29 June 2013. <a href="http://www.playframework.com/documentation/2.1.1/Iteratees" target="_blank">http://www.playframework.com/documentation/2.1.1/Iteratees</a> <a href="#fnref:1" class="footnote-back-ref">↩</a></p></li>
<li id="fn:2"><p>"Github Search Results: Iteratee in FSharpx". GitHub. <a href="https://github.com/fsharp/fsharpx/search?q=iteratee&ref=cmdform" target="_blank">https://github.com/fsharp/fsharpx/search?q=iteratee&ref=cmdform</a> <a href="#fnref:2" class="footnote-back-ref">↩</a></p></li>
<li id="fn:3"><p>"Java theory and practice: The exceptions debate". IBM developerWorks. Retrieved 17 May 2014. <a href="http://www.ibm.com/developerworks/library/j-jtp05254/" target="_blank">http://www.ibm.com/developerworks/library/j-jtp05254/</a> <a href="#fnref:3" class="footnote-back-ref">↩</a></p></li>
<li id="fn:4"><p>"Enumeratees". Play framework documentation. Retrieved 29 June 2013. <a href="http://www.playframework.com/documentation/2.1.1/Enumeratees" target="_blank">http://www.playframework.com/documentation/2.1.1/Enumeratees</a> <a href="#fnref:4" class="footnote-back-ref">↩</a></p></li>
<li id="fn:5"><p>Kiselyov, O. (2012). "Iteratees". Functional and Logic Programming. Lecture Notes in Computer Science. Vol. 7294. pp. 166–181. doi:10.1007/978-3-642-29822-6_15. ISBN 978-3-642-29821-9. <a href="978-3-642-29821-9" target="_blank">978-3-642-29821-9</a> <a href="#fnref:5" class="footnote-back-ref">↩</a></p></li>
<li id="fn:6"><p>James Roper (10 December 2012). "Json.scala". play-iteratees-extras. Retrieved 29 June 2013. <a href="https://github.com/jroper/play-iteratees-extras/blob/master/src/main/scala/play/extras/iteratees/Json.scala" target="_blank">https://github.com/jroper/play-iteratees-extras/blob/master/src/main/scala/play/extras/iteratees/Json.scala</a> <a href="#fnref:6" class="footnote-back-ref">↩</a></p></li>
<li id="fn:7"><p>Kiselyov, O. (2012). "Iteratees". Functional and Logic Programming. Lecture Notes in Computer Science. Vol. 7294. pp. 166–181. doi:10.1007/978-3-642-29822-6_15. ISBN 978-3-642-29821-9. <a href="978-3-642-29821-9" target="_blank">978-3-642-29821-9</a> <a href="#fnref:7" class="footnote-back-ref">↩</a></p></li>
<li id="fn:8"><p>Kiselyov, O. (2012). "Iteratees". Functional and Logic Programming. Lecture Notes in Computer Science. Vol. 7294. pp. 166–181. doi:10.1007/978-3-642-29822-6_15. ISBN 978-3-642-29821-9. <a href="978-3-642-29821-9" target="_blank">978-3-642-29821-9</a> <a href="#fnref:8" class="footnote-back-ref">↩</a></p></li>
</ol>

Iteratee open-in-new

Iteratee