Menu
Home
People
Places
Arts
History
Plants & Animals
Science
Life & Culture
Technology
Reference.org
Snappy (compression)
open-in-new
<h2 id="stream-format">Stream format</h2> <p>Snappy encoding is not bit-oriented, but byte-oriented (only whole bytes are emitted or consumed from a stream). The format uses no <a href="/facts/Entropy_encoding/44I0n8F8">entropy encoder</a>, like <a href="/facts/Huffman_coding/dBgMBz48">Huffman coding</a> or <a href="/facts/Arithmetic_coding/WU0H4oNu">arithmetic coding</a>. </p><p>The first bytes of the stream are the length of uncompressed data, stored as a <a href="/facts/Endianness/wbrQnyu3">little-endian</a> <a href="/facts/Variable-length_quantity/403RnSeL">varint</a>,<a class="footnote-ref" id="fnref:9" href="#fn:9"><sup>9</sup></a>: section 1 which allows for use of a <a href="/facts/Variable-length_code/DImYG6eg">variable-length code</a>. The lower seven bits of each byte are used for data and the high bit is a flag to indicate the end of the length field. </p><p>The remaining bytes in the stream are encoded using one of four element types. The element type is encoded in the lower two bits of the first byte (<i>tag byte</i>) of the element:<a class="footnote-ref" id="fnref:10" href="#fn:10"><sup>10</sup></a> </p> <ul><li>00 – <i>Literal</i> – uncompressed data; upper 6 bits are used to store length (len-1) of data. Lengths larger than 60 are stored in a 1-4 byte integer indicated by a 6 bit length of 60 (1 byte) to 63 (4 bytes).</li> <li>01 – Copy with length stored as 3 bits and offset stored as 11 bits; one byte after tag byte is used for part of offset;</li> <li>10 – Copy with length stored as 6 bits of tag byte and offset stored as two-byte integer after the tag byte;</li> <li>11 – Copy with length stored as 6 bits of tag byte and offset stored as four-byte little-endian integer after the tag byte;</li></ul> <p>The copy refers to the dictionary (just-decompressed data). The offset is the shift from the current position back to the already decompressed stream. The length is the number of bytes to copy from the dictionary. The size of the dictionary was limited by the 1.0 Snappy compressor to 32,768 bytes, and updated to 65,536 in version 1.1. </p><p>The complete official description of the snappy format can be found in the google GitHub repository.<a class="footnote-ref" id="fnref:11" href="#fn:11"><sup>11</sup></a> </p> <h2 id="example-of-a-compressed-stream">Example of a compressed stream</h2> <p>The text </p> <blockquote><p>Wikipedia is a free, web-based, collaborative, multilingual encyclopedia project.</p></blockquote> <p>may be compressed to this, shown as hex data with explanations: </p> 000000 51 <i>f0 42</i> 57 69 6b 69 70 65 64 69 61 20 69 73 20 >Q.BWikipedia is < 000010 61 20 66 72 65 65 2c 20 77 65 62 2d 62 61 73 65 >a free, web-base< 000020 64 2c 20 63 6f 6c 6c 61 62 6f 72 61 74 69 76 65 >d, collaborative< 000030 2c 20 6d 75 6c 74 69 6c 69 6e 67 75 61 6c 20 65 >, multilingual e< <p>The stream starts with the length of the uncompressed data as a varint<a class="footnote-ref" id="fnref:12" href="#fn:12"><sup>12</sup></a>: section 1 – thus the first byte, with the high bit clear, corresponds to a length of 5116=81 bytes. </p><p>The first block must be a literal, and <i>f042</i> corresponds thereto: the first byte is broken down as f016 ⇒ len−1=1111002;type=002; type 0 signifies a literal, and a length−1 of 1111002=60 means the length is read from the following byte, in this case 4216=66. The first 66 bytes of the text ("<i>Wikipedia is a free, web-based, collaborative, multilingual encyclo</i>") follow.<a class="footnote-ref" id="fnref:13" href="#fn:13"><sup>13</sup></a>: 2.1 </p> 000040 6e 63 79 63 6c 6f 09 3f <i>1c</i> 70 72 6f 6a 65 63 74 >ncyclo.?.project< 000050 2e >.< <p>The next block's header consists of 093f, broken down as 0916 ⇒ offh=0002,len−4=0102;type=012: type 1 indicates a "copy with 1-byte offset": the length to copy works out to 0102+4=6 bytes, and the offset is an 11-bit integer whose top bits are offh and whose low bits are the next byte: 3f, so {offh}{3f16}=000001111112=63.<a class="footnote-ref" id="fnref:14" href="#fn:14"><sup>14</sup></a>: 2.2,2.2.1 </p><p>This means to copy 6 bytes, starting 63 bytes ago – since 67 bytes have already been copied this evaluates to copying 6 bytes starting at position 4 (from the fifth byte), which produces "<i>pedia </i>". </p><p>This block has no other content, and thus the following block starts immediately after – <i>1c</i>16 ⇒ len−1=0001112;type=002, i.e. a literal of length 0001112+1=8.<a class="footnote-ref" id="fnref:15" href="#fn:15"><sup>15</sup></a>: 2.1 The final part of the text ("<i>project.</i>") follows. </p><p>In this example, all common substrings with four or more characters were eliminated by the compression process. More common compressors can compress this better. Unlike compression methods such as gzip and bzip2, there is no <a href="/facts/Entropy_encoding/44I0n8F8">entropy encoding</a> used to pack alphabet into the bit stream. </p> <h2 id="framing-format">Framing format</h2> <p>The Snappy stream supports inputs with an overall size of up to 4GiB−1,<a class="footnote-ref" id="fnref:16" href="#fn:16"><sup>16</sup></a>: section 1 and may add significant overhead to sections which are not or insufficiently compressed, as well as not being self-identifying, and having no data integrity mechanism beyond a simple output size check. </p><p>To combat these issues, the Snappy framing format<a class="footnote-ref" id="fnref:17" href="#fn:17"><sup>17</sup></a> "Snappy framed" may be used, which breaks the input into chunks of up to 64KiB,<a class="footnote-ref" id="fnref:18" href="#fn:18"><sup>18</sup></a>: 4.2,4.3 delimited by 4-byte block headers (a one-byte identifier and three-byte length):<a class="footnote-ref" id="fnref:19" href="#fn:19"><sup>19</sup></a>: section 1 </p> <ul><li>the "Stream identifier", with type FF16, must start the stream, and must consist exclusively of "sNaPpY" in <a href="/facts/ASCII/vGUI33Qu">ASCII</a>,<a class="footnote-ref" id="fnref:20" href="#fn:20"><sup>20</sup></a>: 4.1 </li> <li>the "Compressed data", with type 0, contains a compressed Snappy stream,<a class="footnote-ref" id="fnref:21" href="#fn:21"><sup>21</sup></a>: 4.2 </li> <li>the "Uncompressed data", with type 1, contains data to copy to the output verbatim.<a class="footnote-ref" id="fnref:22" href="#fn:22"><sup>22</sup></a>: 4.3 </li></ul> <p>Both types of data chunk also contain a <a href="/facts/CRC-32C/u2exo4sv">CRC-32C</a> checksum of the uncompressed data. </p><p>Chunks of types 2-7F16 are reserved and must result in errors.<a class="footnote-ref" id="fnref:23" href="#fn:23"><sup>23</sup></a>: 4.5 Those of types 8016-FE16 may be ignored by the decompressors which do not understand them.<a class="footnote-ref" id="fnref:24" href="#fn:24"><sup>24</sup></a>: 4.4,4.6 </p> <h2 id="interfaces">Interfaces</h2> <p>Snappy distributions include C++ and C bindings. Third party-provided bindings and ports include<a class="footnote-ref" id="fnref:25" href="#fn:25"><sup>25</sup></a> <a href="/facts/C_Sharp_(programming_language)/UC2gxeyb">C#</a>, <a href="/facts/Common_Lisp/mq1pRbSF">Common Lisp</a>, <a href="/facts/Crystal_(programming_language)/1JgGPFdm">Crystal (programming language)</a>, <a href="/facts/Erlang_(programming_language)/2PuxmzT3">Erlang</a>, <a href="/facts/Go_(programming_language)/oWlsmAFJ">Go</a>, <a href="/facts/Haskell_(programming_language)/4Htv9WLu">Haskell</a>, <a href="/facts/Lua_(programming_language)/xkBCgta0">Lua</a>, <a href="/facts/Java_(programming_language)/9ScgFyAL">Java</a>, <a href="/facts/Nim_(programming_language)/KBmz14Ek">Nim</a>, <a href="/facts/Node.js/FKzhV4Qa">Node.js</a>, <a href="/facts/Perl/fx31kjlT">Perl</a>, <a href="/facts/PHP/vU6YRJX0">PHP</a>, <a href="/facts/Python_(programming_language)/YbuGqofa">Python</a>, <a href="/facts/R_(programming_language)/LSrkr8K8">R</a>, <a href="/facts/Ruby_(programming_language)/BeYSaPSR">Ruby</a>, <a href="/facts/Rust_(programming_language)/8JNMV19F">Rust</a>, <a href="/facts/Smalltalk/Ru2UPVDE">Smalltalk</a>, and <a href="/facts/OpenCL/ns8y6np7">OpenCL</a>.<a class="footnote-ref" id="fnref:26" href="#fn:26"><sup>26</sup></a><a class="footnote-ref" id="fnref:27" href="#fn:27"><sup>27</sup></a> A <a href="/facts/Command-line_interface/R9JpGmXQ">command-line interface</a> program is also available.<a class="footnote-ref" id="fnref:28" href="#fn:28"><sup>28</sup></a> </p> <h2 id="see-also">See also</h2> <ul> <li>Free and open-source software portal</li></ul> <ul><li><a href="/facts/Zstandard/eA9hBQzN">Zstandard</a></li></ul> <h2 id="external-links">External links</h2> <ul><li><a href="https://groups.google.com/forum/#!forum/snappy-compression">Snappy mailing list</a></li></ul> <h2 id="references">References</h2> <ol> <li id="fn:1"><p>"Google Snappy–A Fast Compressing Library". InfoQ. Retrieved August 1, 2011. <a href="http://www.infoq.com/news/2011/04/Snappy" target="_blank">http://www.infoq.com/news/2011/04/Snappy</a> <a href="#fnref:1" class="footnote-back-ref">↩</a></p></li> <li id="fn:2"><p>Google open sources MapReduce compression. In the name of speed // The Register, 2011-03-24 <a href="https://www.theregister.co.uk/2011/03/24/google_open_sources_snappy/" target="_blank">https://www.theregister.co.uk/2011/03/24/google_open_sources_snappy/</a> <a href="#fnref:2" class="footnote-back-ref">↩</a></p></li> <li id="fn:3"><p>"Snappy: A fast compressor/decompressor: Readme". Google Code. Archived from the original on September 8, 2015. Retrieved August 1, 2011."Snappy vs lzo vs zlib". <a href="https://archive.today/20150908105729/https://github.com/google/snappy/blob/master/README" target="_blank">https://archive.today/20150908105729/https://github.com/google/snappy/blob/master/README</a> <a href="#fnref:3" class="footnote-back-ref">↩</a></p></li> <li id="fn:4"><p>"ColumnStore Storage Architecture". MariaDB KnowledgeBase. <a href="https://mariadb.com/kb/en/library/columnstore-storage-architecture/" target="_blank">https://mariadb.com/kb/en/library/columnstore-storage-architecture/</a> <a href="#fnref:4" class="footnote-back-ref">↩</a></p></li> <li id="fn:5"><p>snappy. A fast compressor/decompressor - Project page at Google Code <a href="https://google.github.io/snappy/" target="_blank">https://google.github.io/snappy/</a> <a href="#fnref:5" class="footnote-back-ref">↩</a></p></li> <li id="fn:6"><p>"Compression — Ceph Documentation". Retrieved 2024-01-03. <a href="https://docs.ceph.com/en/latest/radosgw/compression/" target="_blank">https://docs.ceph.com/en/latest/radosgw/compression/</a> <a href="#fnref:6" class="footnote-back-ref">↩</a></p></li> <li id="fn:7"><p>"SnappyUtils.cpp - mozsearch". Retrieved 2024-01-03. <a href="https://searchfox.org/mozilla-central/source/dom/localstorage/SnappyUtils.cpp" target="_blank">https://searchfox.org/mozilla-central/source/dom/localstorage/SnappyUtils.cpp</a> <a href="#fnref:7" class="footnote-back-ref">↩</a></p></li> <li id="fn:8"><p>"Add a loop alignment directive to work around a performance regression. · google/snappy@824e671". GitHub. <a href="https://github.com/google/snappy/commit/824e6718b5b5a50d32a89124853da0a11828b25c" target="_blank">https://github.com/google/snappy/commit/824e6718b5b5a50d32a89124853da0a11828b25c</a> <a href="#fnref:8" class="footnote-back-ref">↩</a></p></li> <li id="fn:9"><p>"Snappy compressed format description". GitHub. 26 October 2021. <a href="https://github.com/google/snappy/blob/main/format_description.txt" target="_blank">https://github.com/google/snappy/blob/main/format_description.txt</a> <a href="#fnref:9" class="footnote-back-ref">↩</a></p></li> <li id="fn:10"><p>"GitHub - google/snappy: A fast compressor/decompressor". November 11, 2019 – via GitHub. <a href="https://github.com/google/snappy" target="_blank">https://github.com/google/snappy</a> <a href="#fnref:10" class="footnote-back-ref">↩</a></p></li> <li id="fn:11"><p>"Snappy compressed format description". GitHub. 26 October 2021. <a href="https://github.com/google/snappy/blob/main/format_description.txt" target="_blank">https://github.com/google/snappy/blob/main/format_description.txt</a> <a href="#fnref:11" class="footnote-back-ref">↩</a></p></li> <li id="fn:12"><p>"Snappy compressed format description". GitHub. 26 October 2021. <a href="https://github.com/google/snappy/blob/main/format_description.txt" target="_blank">https://github.com/google/snappy/blob/main/format_description.txt</a> <a href="#fnref:12" class="footnote-back-ref">↩</a></p></li> <li id="fn:13"><p>"Snappy compressed format description". GitHub. 26 October 2021. <a href="https://github.com/google/snappy/blob/main/format_description.txt" target="_blank">https://github.com/google/snappy/blob/main/format_description.txt</a> <a href="#fnref:13" class="footnote-back-ref">↩</a></p></li> <li id="fn:14"><p>"Snappy compressed format description". GitHub. 26 October 2021. <a href="https://github.com/google/snappy/blob/main/format_description.txt" target="_blank">https://github.com/google/snappy/blob/main/format_description.txt</a> <a href="#fnref:14" class="footnote-back-ref">↩</a></p></li> <li id="fn:15"><p>"Snappy compressed format description". GitHub. 26 October 2021. <a href="https://github.com/google/snappy/blob/main/format_description.txt" target="_blank">https://github.com/google/snappy/blob/main/format_description.txt</a> <a href="#fnref:15" class="footnote-back-ref">↩</a></p></li> <li id="fn:16"><p>"Snappy compressed format description". GitHub. 26 October 2021. <a href="https://github.com/google/snappy/blob/main/format_description.txt" target="_blank">https://github.com/google/snappy/blob/main/format_description.txt</a> <a href="#fnref:16" class="footnote-back-ref">↩</a></p></li> <li id="fn:17"><p>"Snappy framing format description". GitHub. 26 October 2021. <a href="https://github.com/google/snappy/blob/main/framing_format.txt" target="_blank">https://github.com/google/snappy/blob/main/framing_format.txt</a> <a href="#fnref:17" class="footnote-back-ref">↩</a></p></li> <li id="fn:18"><p>"Snappy framing format description". GitHub. 26 October 2021. <a href="https://github.com/google/snappy/blob/main/framing_format.txt" target="_blank">https://github.com/google/snappy/blob/main/framing_format.txt</a> <a href="#fnref:18" class="footnote-back-ref">↩</a></p></li> <li id="fn:19"><p>"Snappy framing format description". GitHub. 26 October 2021. <a href="https://github.com/google/snappy/blob/main/framing_format.txt" target="_blank">https://github.com/google/snappy/blob/main/framing_format.txt</a> <a href="#fnref:19" class="footnote-back-ref">↩</a></p></li> <li id="fn:20"><p>"Snappy framing format description". GitHub. 26 October 2021. <a href="https://github.com/google/snappy/blob/main/framing_format.txt" target="_blank">https://github.com/google/snappy/blob/main/framing_format.txt</a> <a href="#fnref:20" class="footnote-back-ref">↩</a></p></li> <li id="fn:21"><p>"Snappy framing format description". GitHub. 26 October 2021. <a href="https://github.com/google/snappy/blob/main/framing_format.txt" target="_blank">https://github.com/google/snappy/blob/main/framing_format.txt</a> <a href="#fnref:21" class="footnote-back-ref">↩</a></p></li> <li id="fn:22"><p>"Snappy framing format description". GitHub. 26 October 2021. <a href="https://github.com/google/snappy/blob/main/framing_format.txt" target="_blank">https://github.com/google/snappy/blob/main/framing_format.txt</a> <a href="#fnref:22" class="footnote-back-ref">↩</a></p></li> <li id="fn:23"><p>"Snappy framing format description". GitHub. 26 October 2021. <a href="https://github.com/google/snappy/blob/main/framing_format.txt" target="_blank">https://github.com/google/snappy/blob/main/framing_format.txt</a> <a href="#fnref:23" class="footnote-back-ref">↩</a></p></li> <li id="fn:24"><p>"Snappy framing format description". GitHub. 26 October 2021. <a href="https://github.com/google/snappy/blob/main/framing_format.txt" target="_blank">https://github.com/google/snappy/blob/main/framing_format.txt</a> <a href="#fnref:24" class="footnote-back-ref">↩</a></p></li> <li id="fn:25"><p>"snappy". snappy. <a href="http://google.github.io/snappy/" target="_blank">http://google.github.io/snappy/</a> <a href="#fnref:25" class="footnote-back-ref">↩</a></p></li> <li id="fn:26"><p>"Xilinx". Xilinx. <a href="https://github.com/Xilinx/Vitis_Libraries/tree/master/data_compression" target="_blank">https://github.com/Xilinx/Vitis_Libraries/tree/master/data_compression</a> <a href="#fnref:26" class="footnote-back-ref">↩</a></p></li> <li id="fn:27"><p>"InAccel". InAccel. <a href="https://github.com/InAccel/Applications/tree/master/data_compression" target="_blank">https://github.com/InAccel/Applications/tree/master/data_compression</a> <a href="#fnref:27" class="footnote-back-ref">↩</a></p></li> <li id="fn:28"><p>"snappy-tools: snappy(1): Snappy compression and decompression with and without framing". sourcehut. Retrieved 2024-02-15. <a href="https://sr.ht/~nabijaczleweli/snappy-tools/" target="_blank">https://sr.ht/~nabijaczleweli/snappy-tools/</a> <a href="#fnref:28" class="footnote-back-ref">↩</a></p></li> </ol>