Sector/Sphere

<h2 id="architecture">Architecture</h2>
<p>Sector/Sphere consists of four components. The security server maintains the system security policies such as user accounts and the IP access control list. One or more master servers control operations of the overall system in addition to responding to various user requests. The slave nodes store the data files and process them upon request. The clients are the users' computers from which system access and data processing requests are issued. Also, Sector/Sphere is written in <a href="/facts/C%252B%252B/Et0F2qn9">C++</a> and is claimed to achieve with its architecture a two to four times better performance than the competitor <a href="/facts/Hadoop/oUS1G2R7">Hadoop</a> which is written in <a href="/facts/Java_(software)/9ScgFyAL">Java</a>,<a class="footnote-ref" id="fnref:1" href="#fn:1"><sup>1</sup></a> a statement supported by an <a href="/facts/Aster_Data_Systems/KzVK1zBn">Aster Data Systems</a> <a href="/facts/Benchmark_(computing)/XSS42cRe">benchmark</a><a class="footnote-ref" id="fnref:2" href="#fn:2"><sup>2</sup></a> and the winning of the "bandwidth challenge" of the <a href="/facts/Supercomputing_Conference/vtBfdQHj">Supercomputing Conference</a> 2006,<a class="footnote-ref" id="fnref:3" href="#fn:3"><sup>3</sup></a> 2008,<a class="footnote-ref" id="fnref:4" href="#fn:4"><sup>4</sup></a> and 2009.<a class="footnote-ref" id="fnref:5" href="#fn:5"><sup>5</sup></a>
</p>

<h3>Sector</h3>
<p>Sector is a user space file system which relies on the local/native file system of each node for storing uploaded files. Sector provides file system-level fault tolerance by replication, thus it does not require hardware fault tolerance such as <a href="/facts/RAID/IpzWWgog">RAID</a>, which is usually very expensive.
</p><p>Sector does not split user files into blocks; instead, a user file is stored intact on the local file system of one or more slave nodes. This means that Sector has a file size limitation that is application specific. The advantages, however, are that the Sector file system is very simple, and it leads to better performance in Sphere parallel data processing due to reduced data transfer between nodes. It also allows uploaded data to be accessible from outside the Sector system.
</p><p>Sector provides many unique features compared to traditional file systems. Sector is topology aware. Users can define rules on how files are located and replicated in the system, according to network topology. For example, data from a certain user can be located on a specific cluster and will not be replicated to other racks. For another example, some files can have more replicas than others. Such rules can be applied at per-file level.
</p><p>The topology awareness and the use of <a href="/facts/UDP-based_Data_Transfer_Protocol/bqLw0sKC">UDT</a> as data transfer protocol allows Sector to support high performance data IO across geographically distributed locations, while most file systems can only be deployed within a local area network. For this reason, Sector is often deployed as a content distribution network for very large datasets.
</p><p>Sector integrates data storage and processing into one system. Every storage node can also be used to process the data, thus it can support massive in-storage parallel data processing (see Sphere). Sector is application aware, meaning that it can provide data location information to applications and also allow applications to specify data location, whenever necessary.
</p><p>As a simple example of the benefits of Sphere, Sector can return the results from such commands as "grep" and "md5sum" without reading the data out of the file system. Moreover, it can compute the results of multiple files in parallel.
</p><p>The Sector client provides an <a href="/facts/API/GMlN4vUr">API</a> for application development which allows user applications to interact directly with Sector. The software also comes prepackaged with a set of command-line tools for accessing the file system. Finally, Sector supports the <a href="/facts/Filesystem_in_Userspace/RnSLIf1H">FUSE</a> interface; presenting a mountable file system that is accessible via standard command-line tools.
</p>
<h3>Sphere</h3>
<p>Sphere is a parallel data processing engine integrated in Sector and it can be used to process data stored in Sector in parallel. It can broadly compared to <a href="/facts/MapReduce/KbfxJ5vV">MapReduce</a>, but it uses generic user defined functions (UDFs) instead of the map and reduce functions. A UDF can be either a map function or a reduce function, or even others.
Sphere can manipulate the locality of both input data and output data, thus it can effectively support multiple input datasets, combinative and iterative operations and even legacy application executable.
</p><p>Because Sector does not split user files, Sphere can simply wrap up many existing applications that accepts files or directories as input, without rewriting them. Thus it can provide greater compatibility to legacy applications.
</p>
<h2 id="see-also">See also</h2>
<ul>
<li>Free and open-source software portal</li></ul>
<ul><li><a href="/facts/Pentaho/WmDsBlKg">Pentaho</a> - Open source data integration (Kettle), analytics, reporting, visualization and <a href="/facts/Predictive_analytics/tbqzk5qe">predictive analytics</a> directly from Hadoop nodes</li>
<li><a href="/facts/Nutch/6LEseY5n">Nutch</a> - An effort to build an open source search engine based on <a href="/facts/Lucene/tPQ0D39Y">Lucene</a> and Hadoop, also created by Doug Cutting</li>
<li><a href="/facts/Apache_Accumulo/vGCTaaP8">Apache Accumulo</a> - Secure Big Table</li>
<li><a href="/facts/HBase/D4XU1dqK">HBase</a> - <a href="/facts/Bigtable/xlXLKV0t">Bigtable</a>-model database</li>
<li><a href="/facts/Hypertable/w5PYzVJk">Hypertable</a> - HBase alternative</li>
<li><a href="/facts/MapReduce/KbfxJ5vV">MapReduce</a> - Hadoop's fundamental data filtering algorithm</li>
<li><a href="/facts/Apache_Mahout/VzhvrmQL">Apache Mahout</a> - Machine Learning algorithms implemented on Hadoop</li>
<li><a href="/facts/Apache_Cassandra/WCJyV4rA">Apache Cassandra</a> - A column-oriented database that supports access from Hadoop</li>
<li><a href="/facts/HPCC/d1ljmk3V">HPCC</a> - <a href="/facts/LexisNexis/Wu7lgrWK">LexisNexis</a> Risk Solutions High Performance Computing Cluster</li>
<li><a href="/facts/Cloud_computing/guhg0FGV">Cloud computing</a></li>
<li><a href="/facts/Big_data/XGtrVg4z">Big data</a></li>
<li><a href="/facts/Data_Intensive_Computing/BhQMlqq8">Data Intensive Computing</a></li></ul>
<h2 id="literature">Literature</h2>
<ul><li>Yunhong Gu, Robert Grossman, <a href="https://doi.org/10.1098/rsta.2009.0053">Sector and Sphere: The Design and Implementation of a High Performance Data Cloud</a>, Theme Issue of the Philosophical Transactions of the Royal Society A: Crossing Boundaries: Computational Science, E-Science and Global E-Infrastructure, 28 June 2009 vol. 367 no. 1897 2429–2445.</li></ul>

<h2 id="external-links">External links</h2>
<ul><li><a href="http://Sector.sourceforge.net/">Sector/Sphere Project on SourceForge</a></li></ul>

<h2 id="references">References</h2>

<ol>
<li id="fn:1"><p>Sector vs. Hadoop - A Brief Comparison Between the Two Systems <a href="http://sector.sourceforge.net/pub/Sector%20vs%20Hadoop%20-%20v1.pdf" target="_blank">http://sector.sourceforge.net/pub/Sector%20vs%20Hadoop%20-%20v1.pdf</a> <a href="#fnref:1" class="footnote-back-ref">↩</a></p></li>
<li id="fn:2"><p>Sector/ Sphere – Faster than Hadoop/Mapreduce at Terasort September 26, 2010 Ajay Ohri <a href="http://decisionstats.com/2010/09/26/sector-sphere-faster-than-hadoopmapreduce-at-terasort/" target="_blank">http://decisionstats.com/2010/09/26/sector-sphere-faster-than-hadoopmapreduce-at-terasort/</a> <a href="#fnref:2" class="footnote-back-ref">↩</a></p></li>
<li id="fn:3"><p>NCDM Wins Bandwidth Challenge at SC06, HPCWire, November 24, 2006 <a href="https://web.archive.org/web/20091015073135/http://www.hpcwire.com/offthewire/17889209.html" target="_blank">https://web.archive.org/web/20091015073135/http://www.hpcwire.com/offthewire/17889209.html</a> <a href="#fnref:3" class="footnote-back-ref">↩</a></p></li>
<li id="fn:4"><p>UIC Groups Win Bandwidth Challenge Award, HPCWire, November 20, 2008 <a href="https://web.archive.org/web/20091015073216/http://www.hpcwire.com/offthewire/UIC_Groups_Win_Bandwidth_Challenge_Award.html" target="_blank">https://web.archive.org/web/20091015073216/http://www.hpcwire.com/offthewire/UIC_Groups_Win_Bandwidth_Challenge_Award.html</a> <a href="#fnref:4" class="footnote-back-ref">↩</a></p></li>
<li id="fn:5"><p>Open Cloud Testbed Wins Bandwidth Challenge at SC09, December 8, 2009 <a href="https://web.archive.org/web/20120229221150/http://www.hpcwire.com/hpcwire/2009-12-08/open_cloud_testbed_wins_bandwidth_challenge_at_sc09.html" target="_blank">https://web.archive.org/web/20120229221150/http://www.hpcwire.com/hpcwire/2009-12-08/open_cloud_testbed_wins_bandwidth_challenge_at_sc09.html</a> <a href="#fnref:5" class="footnote-back-ref">↩</a></p></li>
</ol>

Sector/Sphere open-in-new

Sector/Sphere