Menu
Home
People
Places
Arts
History
Plants & Animals
Science
Life & Culture
Technology
Reference.org
Standard Compression Scheme for Unicode
open-in-new
<h2 id="history--use">History & use</h2> <p><a href="/facts/Reuters/tDd1EUrd">Reuters</a> originally developed SCSU, then under the name RCSU for Reuters Compression Scheme for Unicode.<a class="footnote-ref" id="fnref:2" href="#fn:2"><sup>2</sup></a><a class="footnote-ref" id="fnref:3" href="#fn:3"><sup>3</sup></a><a class="footnote-ref" id="fnref:4" href="#fn:4"><sup>4</sup></a><a class="footnote-ref" id="fnref:5" href="#fn:5"><sup>5</sup></a> </p><p>At first the Unicode Consortium considered it to be a character encoding,<a class="footnote-ref" id="fnref:6" href="#fn:6"><sup>6</sup></a> but in 1999 changed its mind: although it was still considered a transfer encoding syntax, for a while it was no longer considered a character encoding because different compressors might yield different outputs for the same text.<a class="footnote-ref" id="fnref:7" href="#fn:7"><sup>7</sup></a> However, in 2004 this decision was reverted and now SCSU is considered a <i>compressing</i> character encoding scheme, as opposed to a simple or compound character encoding scheme.<a class="footnote-ref" id="fnref:8" href="#fn:8"><sup>8</sup></a><a class="footnote-ref" id="fnref:9" href="#fn:9"><sup>9</sup></a> </p><p>Roman Czyborra (of <a href="/facts/GNU_Unifont/I9S5eXEx">GNU Unifont</a>) wrote a decompressor.<a class="footnote-ref" id="fnref:10" href="#fn:10"><sup>10</sup></a> The IBM-contributed decompressor is found in <a href="/facts/International_Components_for_Unicode/SVyD2Yr5">International Components for Unicode</a>, along with a compressor written in Java.<a class="footnote-ref" id="fnref:11" href="#fn:11"><sup>11</sup></a> Simpler reference codecs are available as attachments to TR6. </p><p><a href="/facts/Symbian_OS/gbRl35V9">Symbian OS</a>, an operating system for mobile phones and other mobile devices, uses SCSU to serialize strings. </p><p><a href="/facts/Microsoft_SQL_Server/eCdt2JBL">SQL Server 2008 R2</a> uses SCSU to compress Unicode values (there meaning from strings in <a href="/facts/UCS-2/7Tn6LPLw">UCS-2</a> encoding) stored in <i>nchar(n)</i> and <i>nvarchar(n)</i> columns, achieving space savings between 15% and 50% (while <a href="/facts/UTF-8/xlaVHSyf">UTF-8</a> only has this 50% reduction for <a href="/facts/ASCII/vGUI33Qu">ASCII</a> subset of Unicode), depending on the language of the data.<a class="footnote-ref" id="fnref:12" href="#fn:12"><sup>12</sup></a> </p> <h2 id="the-scheme">The scheme</h2> <p>The following sections briefly describe the anatomy of a compressed SCSU stream. For a full description (matching that of a decompressor), see the UTS #6 document. </p> <h3>Encoding modes</h3> <p>SCSU starts in the single-byte mode, which uses the compressed Window encoding. There exist commands to switch to a UTF-16BE "Unicode" mode, and to switch to the single-byte mode from that mode. </p> <h3>Window encoding</h3> <p>The core of SCSU lies in the windows for which the meanings of bytes 0x80-0xff are defined. There are eight static windows for simpler scripts and punctuation, and 6 types of dynamic windows (plus "half Unicode block" windows and custom Windows for the supplementary planes) for scripts making use of more characters. </p><p>Both simple and dynamic windows are selected by special command characters. For individual characters that do not fit into the current block, command characters for quoting are provided. </p> <h2 id="comparison-with-general-purpose-plain-text-compression-schemes">Comparison with general-purpose plain text compression schemes</h2> <p>Because UTF-16 or UTF-8 text might occupy more space than its equivalent in pre-Unicode encodings did, one might want to use compression such as SCSU to mitigate this problem.<a class="footnote-ref" id="fnref:13" href="#fn:13"><sup>13</sup></a> In comparison with general-purpose compressors, it is not necessarily advantageous to use SCSU.<a class="footnote-ref" id="fnref:14" href="#fn:14"><sup>14</sup></a> Also, while it can be used as a text encoding, because of the stateful nature of the algorithm difficulties may arise when using it as an internal text representation since basic text operations become non-trivial. </p><p>Treated purely as a compression algorithm, SCSU is inferior to most commonly used general-purpose algorithms for texts of over a few kilobytes. </p><p>SCSU does have the advantage that it can usefully compress texts that are only a few characters long, whereas most full-scale compressors need hundreds of bytes of data to break even against their own overhead. In <a href="/facts/Symbian_OS/gbRl35V9">Symbian OS</a>, SCSU is used even for Clipboard operations, e.g. Cut, Copy & Paste of small strings of text. </p> <h2 id="in-html">In HTML</h2> <p>Supporting SCSU in <a href="/facts/HTML/sSc3fT78">HTML</a> documents is prohibited by the <a href="/facts/W3C/lGWmhF6M">W3C</a><a class="footnote-ref" id="fnref:15" href="#fn:15"><sup>15</sup></a><a class="footnote-ref" id="fnref:16" href="#fn:16"><sup>16</sup></a> and <a href="/facts/WHATWG/WoyKwRJR">WHATWG</a><a class="footnote-ref" id="fnref:17" href="#fn:17"><sup>17</sup></a> HTML standards because HTML wasn't designed with non-ASCII-compatible encodings in mind. In the past, <a href="/facts/Cross-site_scripting/dhtusKKm">cross-site scripting</a> vulnerabilities due to browsers' poor handling of such encodings have been demonstrated.<a class="footnote-ref" id="fnref:18" href="#fn:18"><sup>18</sup></a> </p> <h2 id="see-also">See also</h2> <ul><li><a href="/facts/Binary_Ordered_Compression_for_Unicode/oL4BHfCs">Binary Ordered Compression for Unicode</a> (BOCU-1)</li> <li><a href="/facts/International_Components_for_Unicode/SVyD2Yr5">International Components for Unicode</a> A library that can convert between SCSU and other Unicode encodings</li></ul> <h2 id="references">References</h2> <ol> <li id="fn:1"><p>"UTS #6: Compression Scheme for Unicode". 2005-05-06. Retrieved 2008-06-13. SCSU defines a compact encoding, which is sometimes useful. However, Unicode text is much more commonly stored and transmitted in UTF-8 which is less compact (except for ASCII), much simpler, and does not present any security issues. For longer texts, general-purpose compression is effective and common. <a href="https://www.unicode.org/reports/tr6/" target="_blank">https://www.unicode.org/reports/tr6/</a> <a href="#fnref:1" class="footnote-back-ref">↩</a></p></li> <li id="fn:2"><p>"Ninth International Unicode Conference - Friday - Track B". <a href="https://unicode.org/iuc/iuc9/Friday2.html#b3" target="_blank">https://unicode.org/iuc/iuc9/Friday2.html#b3</a> <a href="#fnref:2" class="footnote-back-ref">↩</a></p></li> <li id="fn:3"><p>"Tenth International Unicode Conference - Conference Program". <a href="https://unicode.org/iuc/iuc10/program.html" target="_blank">https://unicode.org/iuc/iuc10/program.html</a> <a href="#fnref:3" class="footnote-back-ref">↩</a></p></li> <li id="fn:4"><p>"Compression Scheme for Unicode". <a href="https://unicode.org/reports/tr6-10.html" target="_blank">https://unicode.org/reports/tr6-10.html</a> <a href="#fnref:4" class="footnote-back-ref">↩</a></p></li> <li id="fn:5"><p>"A survey of Unicode compression" (PDF). <a href="https://www.unicode.org/notes/tn14/UnicodeCompression.pdf" target="_blank">https://www.unicode.org/notes/tn14/UnicodeCompression.pdf</a> <a href="#fnref:5" class="footnote-back-ref">↩</a></p></li> <li id="fn:6"><p>"UTR#17: Character Encoding Model". <a href="https://unicode.org/reports/tr17/tr17-2.html" target="_blank">https://unicode.org/reports/tr17/tr17-2.html</a> <a href="#fnref:6" class="footnote-back-ref">↩</a></p></li> <li id="fn:7"><p>https://unicode.org/reports/tr17/tr17-3.html#Transfer Encoding Syntax <a href="https://unicode.org/reports/tr17/tr17-3.html#Transfer" target="_blank">https://unicode.org/reports/tr17/tr17-3.html#Transfer</a> <a href="#fnref:7" class="footnote-back-ref">↩</a></p></li> <li id="fn:8"><p>"UTR#17: Character Encoding Model". 2004-07-14. <a href="https://unicode.org/L2/L2004/04288-tr17-5d2.html#CharacterEncodingScheme" target="_blank">https://unicode.org/L2/L2004/04288-tr17-5d2.html#CharacterEncodingScheme</a> <a href="#fnref:8" class="footnote-back-ref">↩</a></p></li> <li id="fn:9"><p>"UTR#17: Unicode Character Encoding Model". unicode.org. Retrieved 2023-11-14. <a href="https://unicode.org/reports/tr17/" target="_blank">https://unicode.org/reports/tr17/</a> <a href="#fnref:9" class="footnote-back-ref">↩</a></p></li> <li id="fn:10"><p>"This is a deflator to UTF-8 output for input compressed in SCSU". Archived from the original on 1999-09-08. <a href="https://web.archive.org/web/19990908230458/http://czyborra.com:80/scsu/scsu.c" target="_blank">https://web.archive.org/web/19990908230458/http://czyborra.com:80/scsu/scsu.c</a> <a href="#fnref:10" class="footnote-back-ref">↩</a></p></li> <li id="fn:11"><p>"International Components for Unicode". GitHub. 22 October 2021. <a href="https://github.com/unicode-org/icu/blob/3f043c7693e20c8cded76035918dad104e7256e3/icu4j/main/classes/charset/src/com/ibm/icu/charset/CharsetSCSU.java" target="_blank">https://github.com/unicode-org/icu/blob/3f043c7693e20c8cded76035918dad104e7256e3/icu4j/main/classes/charset/src/com/ibm/icu/charset/CharsetSCSU.java</a> <a href="#fnref:11" class="footnote-back-ref">↩</a></p></li> <li id="fn:12"><p>"Unicode Compression Implementation (SQL Server 2008 R2 Books Online)". Retrieved 2008-08-18. <a href="https://docs.microsoft.com/en-us/previous-versions/sql/sql-server-2008-r2/ee240835(v=sql.105)" target="_blank">https://docs.microsoft.com/en-us/previous-versions/sql/sql-server-2008-r2/ee240835(v=sql.105)</a> <a href="#fnref:12" class="footnote-back-ref">↩</a></p></li> <li id="fn:13"><p>"Implementation Guidelines" (PDF). Archived from the original (PDF) on 2015-07-30. <a href="https://web.archive.org/web/20150730234318/http://www.unicode.org/versions/Unicode3.0.0/ch05.pdf" target="_blank">https://web.archive.org/web/20150730234318/http://www.unicode.org/versions/Unicode3.0.0/ch05.pdf</a> <a href="#fnref:13" class="footnote-back-ref">↩</a></p></li> <li id="fn:14"><p>"A survey of Unicode compression" (PDF). <a href="https://www.unicode.org/notes/tn14/UnicodeCompression.pdf" target="_blank">https://www.unicode.org/notes/tn14/UnicodeCompression.pdf</a> <a href="#fnref:14" class="footnote-back-ref">↩</a></p></li> <li id="fn:15"><p>"8.2.2.3. Character encodings". HTML 5.1 Standard. W3C. <a href="https://www.w3.org/TR/html51/syntax.html#character-encodings" target="_blank">https://www.w3.org/TR/html51/syntax.html#character-encodings</a> <a href="#fnref:15" class="footnote-back-ref">↩</a></p></li> <li id="fn:16"><p>"8.2.2.3. Character encodings". HTML 5 Standard. W3C. <a href="https://www.w3.org/TR/html5/syntax.html#character-encodings" target="_blank">https://www.w3.org/TR/html5/syntax.html#character-encodings</a> <a href="#fnref:16" class="footnote-back-ref">↩</a></p></li> <li id="fn:17"><p>"12.2.3.3 Character encodings". HTML Living Standard. WHATWG. <a href="https://html.spec.whatwg.org/multipage/parsing.html#character-encodings" target="_blank">https://html.spec.whatwg.org/multipage/parsing.html#character-encodings</a> <a href="#fnref:17" class="footnote-back-ref">↩</a></p></li> <li id="fn:18"><p>"<meta> - HTML". MDN Web Docs. Mozilla. Archived from the original on 3 October 2018. <a href="https://web.archive.org/web/20181003090922/https://developer.mozilla.org/en-US/docs/Web/HTML/Element/meta" target="_blank">https://web.archive.org/web/20181003090922/https://developer.mozilla.org/en-US/docs/Web/HTML/Element/meta</a> <a href="#fnref:18" class="footnote-back-ref">↩</a></p></li> </ol>