Name | Country | Creation Year | Technologies | Number of Employees | Comments |
---|
Full-time | Part-time |
---|
End of Term Web Archive | United States | 2008 | Heritrix, Wayback | | 6–10 | The End of Term Web Archive captures and saves U.S. Government federal government websites (.gov, .mil, etc) in the Legislative, Executive, or Judicial branches of the government at the end of presidential administrations. Beginning in 2008, the EOT has thus far preserved websites from administration changes in 2008, 2012 and 2016, and is currently gearing up for the 2020 transition. Project partners include CA Digital Library, Internet Archive, Library of Congress, George Washington University, Stanford University, University of North Texas, and the US Government Publishing Office. |
Arkiwera | Worldwide (but based in Sweden) | 2020 | Open source solutions and custom programming and scripts | 3 | 4 | Arkiwera is a Swedish company that maintains digital archives of websites and social media accounts for an annual fee. It supports automatic collection, replay, full-text search and data exports. |
EU Web Archive | European Union | 2013 | Archive-it service | 1 | | The EU Web Archive compiles the captures of the websites of the European Union institutions, which are hosted on the europa.eu domain and subdomains. Its aim is to preserve EU web content in the long term and to keep it accessible for the public. The archive was created in 2013 by the Historical Archives of the European Union and in 2018, the Publications Office of the EU took over this task and created the EU Web Archive service. The collection of archived websites is covered by the EU Legal Deposit scheme, which collects all the material produced by EU entities in a comprehensive bibliography. |
Alabama State Government and Politics Web Site and Social Media Archives | United States | 2005 | Archive-it service | | | |
Australia's Web Archive | Australia | 1996 | PANDORA Digital Archiving System (PANDAS), Heritrix, Bamboo, NLA Trove, HTTrack, Webrecorder, outbackCDX. | 4 | >10 | The National Library of Australia leads the 'PANDORA' component of the Australian Web Archive which takes a selective approach and is a collaborative program of 10 agencies providing curatorial input. PANDORA uses the PANDAS workflow system (developed by the NLA in the late 1990s) with HTTrack as the default harvester. The National Library of Australia also conducts bulk harvesting of Australian government (the Australian Government Web Archive) websites using the Heritrix harvester and Webrecorder with a backend infrastructure (referred to as 'Bamboo') to organise content and the NLA developed outbackCDX tool to manage indexing access restrictions for content. In addition to these approaches the National Library also conducts annual harvests of the whole .au domain which is done in collaboration with the Internet Archive using Heritrix and Wayback. In 2019, PANDORA, the Australian Government Web Archive and the whole domain harvests were integrated into a new single discovery and delivery portal through the NLA's Trove discovery service. |
PROMISE project | Belgium | 2017 | Heritrix, PyWB | | 7 | The PROMISE project was a two-year project (2017–2019) that explored the policy-related, legal, technical and scientific issues related to archiving the Belgian web. The aim of the project was to a) identify best practices in the field of web-archiving b) develop a strategy for preserving the Belgian web c) set up a pilot for preserving and providing access to the archived Belgian web and d) make recommendations for the implementation of a sustainable web-archiving service. The project was launched by the Royal Library of Belgium and the State Archives of Belgium in collaboration with Ghent University (Research Group for Media, Innovation and Communication and Ghent Centre for Digital Humanities), Université de Namur (Research Centre in Information, Law and Society) and Haute-École Bruxelles-Brabant (Unité de Recherche et de Formation en Sciences de l'Information et de la Documentation). In October 2019 the concluding colloquium 'Saving the web: the promise of a Belgian web archive') took place at KBR. The main research findings were presented during this colloquium. |
KBR web archive | Belgium | 2020 | | 1 | | KBR or the Belgian Royal Library is developing an operational web archive based on the findings of the PROMISE research project (2017–2019). Operational policies and technical infrastructure will be developed based on the strategy outlined in the PROMISE project. |
KADOC-KU Leuven | Belgium | 2022 | HTTrack, Heritrix, Archiveweb.page, Replayweb.page | 0 | 1 | Thematic archive with a collection concentrated around the interaction between religion, culture and society in Belgium. In 2023 a research project Best practices for social media archiving in Flanders and Brussels ended. |
MT.GOV Connect | United States | 2007 | Archive-It Service | 1 | | Montana State Library collection of state agency websites dating from 1996 in partial fulfillment of statutory mandate to identify, acquire, describe, and provide permanent public access to state publications. Digitized historic state publications available at https://archive.org/details/MontanaStateLibrary |
Stillio | Worldwide | 2011 | Puppeteer, V8 engine, Gecko, WebKit, Amazon Web Services | 3 | 4 | SaaS solution for periodical website & social media archiving. Provides screenshot archiving of both static and dynamic web pages in a fixed duration which can be customized as per requirements. Helps in regulatory compliances, trend tracking, ad banner verification, version changes. |
PageFreezer | Worldwide | 2009 | PageFreezer's Deep Web Crawler, Hadoop, Cassandra, Elastic Search | 60 | | SaaS solution for website & social media archiving. Provides automatic collection, replay, full-text search and data export of websites, blogs, social media and enterprise collaboration platforms for eDiscovery and regulatory compliance with FDA, FINRA, FSA, SEC, Federal Rules of Evidence, FOIA and records management laws. |
OoCities — GeoCities Archive / GeoCities Mirror | Germany | 2009 | | | | |
Wikiwix Archive — Linterweb | France | 2008 | Selenium + MongoDB | | | In production on French-speaking Wikipedia since 2008, open-source project which optimizes the consumption of inodes and thus fills hard drives. Contains an annotation space for archived documents. Main developer Fabien Coulon doctor du Litis on behalf of Linterweb, hosted by Renater https://gitlab.com/dev_linterweb. |
Webarchive Austria | Austria | 2008 | NetarchiveSuite, Heritrix, OpenWayback | 1 | 1 | |
Deutsche Nationalbibliothek | Germany | 2012 | Tools of oia GmbH | | 6 | The crawling for the selective web archive is done by the German company oia GmbH. The access is restricted to the reading rooms of the German National Library. |
DILIMAG (Digital Literature Magazines) | Austria | 2007 | WebCurator | | 2 | One technician, one for collecting and metadata. |
Bibliothèque et Archives nationales du Québec (BAnQ) | Canada | 2012 | Heritrix, Wayback. | | 2 | |
Web Archiving Program at Library and Archives Canada | Canada | 2005 | Archive-It service | 4 | 3 | Web archiving in Canada is a legislated activity that is conducted for digital preservation purposes under section 8 (2) of the Library and Archives of Canada Act. Four FTEs and three part-time staff work on the program. Web archiving at Library and Archives Canada is also utilized to effect Legal Deposit. |
Web Information Collection and Preservation - WICP (Chinese Web Archive) | China | 2003 | Heritrix, Wayback and NutchWAX Archived 2015-06-26 at the Wayback Machine. | | | |
Croatian Web Archive (Hrvatski arhiv weba - HAW) | Croatia | 2004 | Crawl: DAMP software, Heritrix Access: Wayback, Lucene | 2 | 2 | The Croatian Web Archive (HAW) is a collection of content harvested from the Internet. In 2004 the Archive started as a concept of selective capturing of web resources. Whole .hr domain harvests have been conducted annually since 2011. as well as thematic/event harvesting for events of national interest. The content of the Archive is publicly available via HAW website. (2 librarians full time, 1 librarian part time, NUL), 2 IT professionals part time (SRCE - University of Zagreb, University Computing Centre) |
Webarchiv (National Library of the Czech Republic) | Czech Republic | 2000 | Heritrix, Wayback and Seeder. | 5 | 2 | Czech web archive (Webarchiv) maintained by National Library of the Czech Republic focuses on archiving the Czech national web. Acquisition policy consists of three lines: selective harvests (collection of resources based on selection criteria), topic collections (focused on significant topics in the area of the Czech web) and comprehensive harvests (automatic harvests of content on the national domain). Staff contains 1 manager, 3.5 curators + 1.5 technical staff. |
Netarkivet/ The Danish web archive (Royal Danish Library) | Denmark | 2005 | Schedule/crawling: NetarchiveSuite, Heritrix, Browsertrix, Archiveweb.page Access/search/discovery frontend and playback: SolrWayback. Still installed Wayback for alternative playback, but planning to migrate to PyWb. | 1 | 5.5 FTE | Since 2005 the collection and preservation of the Danish part of the internet is included in the Danish Legal Deposit Law. The task is undertaken by the Royal Danish Library. There is no public access to the Danish web archive .The archive is only accessible to researchers affiliated with a Danish research institution who have requested and been granted special permission to use the collection for specific research purposes. This website https://www.kb.dk/en/find-materials/collections/netarkivet is designed to inform researchers, website owners, and other interested parties about the Danish web archive. |
Estonian Web Archive | Estonia | 2010 | Heritrix, Squidwarc, PhantomJS and Puppeteer for screenshots of websites frontpages, Pywb, Custom Curator Tool. | 3 | 1 | Since 2006 the Legal Deposit Law allows the National Library of Estonia to collect Estonian websites as legal deposit copies. Web harvesting is done and archive is maintained by the National Library of Estonia. |
Finnish Web Archive | Finland | 2006 | Heritrix, Solr, Pywb, Browsertrix crawler, Webrecorder -addon, OutbackCDX, Twarc2, YT-DPL. | 3 | >3 | Maintained by the National Library of Finland. Annually, all *.fi domains are harvested, as well as web servers located in Finland. Outside these harvests, the library manually selects relevant websites. |
BnF - Web Legal Deposit | France | 2006 | Heritrix, NetarchiveSuite, BCWeb, OpenWayback, SolrWayback, WARC Indexer/Solr | 11 | | In France, since 2006, the law on copyright and related rights in the information society (known as DADVSI) extended the scope of legal deposit to "signs, signals, writings, images, sounds or messages of any kind " communicated to the public by electronic means - in other words legal deposit of the web. Archiving the French web is a legal commitment, which continues the heritage mission of the BnF. As it is technically impossible to permanently collect all Web content, the goal of completeness from the legal deposit of printed documents has evolved into a sampling approach to create digital collections that show the production and the behaviour of French internet users. |
Ina (Institut National de l'Audiovisuel) | France | 2009 | Crawl: PhagoSite, Crocket based on Firefox, Fantomas based on PhantomJS / Access: Vortex / Search: Dowser based on Elasticsearch | 7 | | |
Bibliotheksservice-Zentrum Baden-Württemberg | Germany | 2003 | Archive-It service | | 0.5 | Websites of about 20 cities, municipalities, districts and associated corporations, and state libraries are collected by BSZ in commission within various Archive-It collections. Public access. Data storage: San Francisco (Archive-It) as well as backup with Baden-Wuerttemberg storage infrastructure. |
Web archive of the German Bundestag | Germany | 2005 | | | | |
National Széchényi Library Web Archive | Hungary | 2017 | Heritrix, Wayback, PyWb, Brozzler, Webrecorder, WCT | 3 | 2 | From April 2017 till December 2019 the National Széchényi Library (http://www.oszk.hu) ran a web archiving pilot project as part of its comprehensive IT infrastructure development programme. In 2020 web archiving became a permanent service of the National Széchényi Library. From 2021 on, the legal framework was established and the web archive works according to the modified paragraphs of the cultural law and the corresponding government decree. They run thematic, event-based and domain harvests. They have a small demo collection with metadata and full-text search capabilities. The rest of the archive is not publicly available. |
Iceland | Iceland | 2004 | Heritrix, OpenWayback | | | |
National Library of Ireland Web Archive | Ireland | 2011 | Archive-it service | 1 | 0.5 FTE | The National Library of Ireland selectively archives Irish websites of scholarly, cultural and political importance through its NLI Selective Web Archive. |
Palestine Web Archive | Palestine | 2011 | Heritrix, Web curator tool, Wayback, Rosetta | 1 | >3 | National Library of Palestine collecting '.PS' domains, 1 Project Manager part time, 1 Technical Leader full time, 1 librarian part time, 1 IT Infrastructure part time |
National Central Library of Florence | Italy | 2018 | Archive-it Service | | | The aim of the project is to collect and to archive digital documents and websites having "cultural interest" for Italian history and culture, according with the principles of the national legal deposit law. The Archive-it Collection is publicly available. |
Web Archiving Project (WARP), The National Diet Library, Japan | Japan | 2002 | Heritrix, OpenWayback, Solr | 4 | 1 | Web Archiving Project (WARP) has been archiving websites since 2002. The National Diet Library Law revised in 2009 and coming into force in April 2010, allows the NDL to archive Japanese official institutions' websites: the government, the Diet, the courts, local governments, independent administrative organizations, and universities. Websites of cultural and international events held in Japan, and those related to online periodicals, are also archived based on the permission of their webmasters. |
National Library of Korea - OASIS (Online Archiving & Searching Internet Sources) | Korea | 2001 | Own system based on Oracle DBMS and specialized search engine (IRS) that performs data management and search function. | 3 | 11 | |
Bibliothèque nationale du Luxembourg | Luxembourg | 2015 | Heritrix, Wayback, Browsertrix, Solr | 2 | | The National library of Luxembourg conducts quarterly broad crawls for the .lu domain as well as selective and event-based crawls. The websites that are harvested in the Luxembourg Web Archive enrich the patrimonial collections of the National library, which allows for the preservation of digital publications for future generations. Webarchive.lu is the Luxembourg Web Archive's information and participation platform. |
Koninklijke Bibliotheek | Netherlands | 2007 | Heritrix 3.3, Web Curator Tool 3.0, Wayback, KB e-Depot system | | ~10 | 1 crawl engineer, 1 software developer, and 9 collection specialists, all part-time (equivalent to around 4 full-time). The KB selectively collects Dutch sites of research and cultural value. |
National Library of Latvia | Latvia | 2005 | Web Curator Tool and Wayback | | 1 | Currently only storing for preservation, access to public in development (ETA June 2012). The Latvian term for web harvesting is "rasmošana". |
New Zealand Web Archive | New Zealand | 1999 | Web Curator Tool, Heritrix3, Webrecorder, ArchiveIT, Browsertrix, Pywb, OutbackCDX, Rosetta | 5 | >10 | National domain harvests have been run since 2008, and annually since 2015 in collaboration with the Internet Archive. Selective harvesting is undertaken by the National Library of New Zealand primarily using the Web Curator Tool. Three full time staff harvest websites and a number of rostered staff harvest HTML serials or HTML monographs. Supported by one dedicated web archiving engineer, and wider departmental ITMS. Digital Preservation issues are handled by staff who work with Rosetta. |
The National Library of Norway | Norway | 2001 | | | | |
Arquivo.pt | Portugal | 2007 | In-house development, Heritrix, Wayback, NutchWAX Archived 2015-06-26 at the Wayback Machine, Pywb, Apache Solr, Brozzler, Webrecorder.net tools | 3 | 4 | Arquivo.pt is a research infrastructure that preserves information gathered from the web since 1996 and provides a public search service over this collection. Arquivo.pt preserves websites in several languages and provides user interfaces in English. The archived data can be automatically processed to perform Big Data research through a distributed processing platform or through Application Programming Interfaces that facilitate the development of added-value applications. The Arquivo.pt team has also contributed with scientific and technical articles related to web archiving published in open-access. |
Web archive of Cacak | Serbia | 2009 | HTTrack | | 1 | |
Web Archive Singapore | Singapore | 2006 | Wayback, Heritrix, Solr | 3 | | The Web Archive Singapore is managed by the National Library Board, Singapore (NLB). NLB conducts domain and selective archiving of websites with a focus on Singapore content. The collection is viewable at the National Library, Singapore with selected content cleared by copyright owners available online. |
Digital Resources (University Library in Bratislava) | Slovak Republic | 2015 | Heritrix 3.2.0, OpenWayback 2.2.0, Solr 5.2.1, Invenio, Custom Curator Tool, Archivewebpage.org | 4 | 1 | The University Library in Bratislava (ULIB) performed the first experiments of webharvesting in 2008–2009. In 2015 ULIB carried into operation a platform for web- and e-Born archiving (during the implementation of the national project "Digital resources", that was supported by the European regional development fund) - www.webdepozit.sk/). |
Slovenian Web Archive | Slovenia | 2007 | Heritrix, OpenWayback, Web Curator Tool | 1 | | |
Archivo de la Web Española | Spain | 2009 | NetarchiveSuite, OpenWayback, Solr | 3+supervisor | 2 | Maintained by the National Library of Spain with the collaboration of regional libraries. Takes a mixed approach of selective and broad harvests. Whole .es domain harvests have been conducted annually since 2009 to 2013 in collaboration with the Internet Archive using Heritrix and Wayback. Since 2014 selective harvests have been made by National Library of Spain, using NetarchiveSuite. National Library = 3 librarians full time, 2 crawl engineers part time. Regional libraries = several librarians part time. Since 26 October 2015 the Legal Deposit Law allows the National Library of Spain and the regional libraries to collect Spanish websites as part of the legal deposit and make them available to the public observing the rules of copyright law. |
PADICAT: The Web Archive of Catalonia | Spain | 2005 | Heritrix, OpenWayback, OutbackCDX and CAT. | | 2 | PADICAT is the open access Web Archive of Catalonia, created by the Biblioteca de Catalunya: the public institution responsible for collecting, preserving and distributing the bibliographic heritage of Catalonia, in Spain. |
ONDARENET - Basque Digital Heritage Archive | Spain | 2008 | Heritrix, Wayback, NutchWAX Archived 2015-06-26 at the Wayback Machine and Web Curator. | 1 | | |
Sweden (Kulturarw3) | Sweden | 1996 | NetarchiveSuite, Heritrix. Inhouse system for storage, maintenance and access, but moving to pywb or SolrWayback. | | 1.25 | The Swedish web harvesting project started in 1996 and the first harvest was performed in 1997. In 2002 daily harvests of certain newspaper web sites were added. There was a pause in operation November 2009 - May 2011, but a harvest for 2010 was made with the help of the Internet Archive. No domain harvests were made in 2016, 2018 and 2019 due to problems with the harvesting platform. The daily harvests of newspaper websites were paused between May 2017 and December 2018, but was the expanded to cover all Swedish newspaper web sites on a daily basis. Since April 2013 the National Library of Sweden also receives online material through the Legal Deposit Act for Electronic Material. |
Aleph Archives | Switzerland, United States | 2010 | Web archiving platform, capture domain name, high performance search engine, Near real time indexing, Web Monitoring tools | >10 | | Enterprise-grade automatic web archiving platform for online capture and preservation. Support eDiscovery with powerful and qualitative technology. Aimed to corporations, institutions and agencies seeking to capture, preserve and leverage their Web content; dynamic websites, wikis, social media, forums, comments, disclaimers, and ads, for compliance (FDA, FINRA, FSA, SEC, FOIA), marketing or pure preservation purposes. |
Expatriate Archive Centre Blog Archive | The Hague, The Netherlands | 2019 | Archive-It service | | | The focus of this project is blogs written by any people who have lived abroad. We preserve these blogs and their contents because we recognise their cultural and historical value. Adding a blog archive to our collection will enrich the research opportunities for students and other academics who choose the us as a place of study. The archived blogs will be selected based on very specific criteria and their quality will be checked on a regular basis. |
Web Archiving Bucket | Switzerland, United States, Canada | 2012 | WARC Software Development Kit, Cobalt, Holon web server | | | The "Web Archiving Bucket" is an initiative launched by Aleph Archives, to preserve data and provide libraries and organizations with free-to-use web archiving tools and components. The Web Archiving Bucket provides set of tools to help archivists and professionals in their daily work. |
Web Archive Switzerland | Switzerland | 2008 | Heritrix, Wayback, Pywb, Webrecorder, Browsertrix Cloud | | 6 | 2 crawl engineers, 3 persons for quality assurance (sharing less than 1 full time), 1 coordinator. The curators, who do the selection, are partner libraries all over Switzerland. |
NTU Web Archiving System, NTUWAS | Taiwan | 2007 | Lucene | | 3 | |
Web Archive Taiwan | Taiwan | 2007 | | | | |
UK Web Archive | United Kingdom | 2004 | Heritrix, Web Curator Tool, Wayback, Solr for searching. | | | |
UK Government Web Archive (UKGWA) | United Kingdom | 2003 | MirrorWeb | 7 | 1 | The UK National Archives' UK Government Web Archive (UKGWA) is a fully open web archive. It includes over 5,000 central government websites and social media taken at regular intervals (1996 to present). The scope of UKGWA is outlined in the OSP27 document. Technical side of web archiving operation is supplied by MirrorWeb. |
UK Parliament Web Archive | United Kingdom | 2009 | MirrorWeb | 1 | 2 | The UK Parliament Web Archive captures, preserves, and make accessible UK Parliament information published on the web. The web archive includes websites and social media dating from 2009 to the present. The technical side of web archiving operation is supplied by MirrorWeb. |
EU Exit Web Archive | United Kingdom | 2020 | MirrorWeb | | | The UK National Archives' EU Exit Web Archive is a fully open web archive. It contains a wide selection of documents taken from EUR-Lex (the European legislation website), including Treaties, legislative documents, the Official Journal of the EU, case law and other supporting materials, and judgements of the European Court of Justice in English, French and German. The collection contains all content published up to the completion of the implementation period, at 11pm GMT on 31 December 2020. It provides a comprehensive and official UK reference point for EU law as it stood at the end of the implementation period. The technical side of web archiving operation is supplied by MirrorWeb. |
MirrorWeb | Worldwide | 2012 | Heritrix, PYWB for public archives, custom replay for archives inside the MirrorWeb platform. Custom social media archiving tools. | 40 | | MirrorWeb provides a website and social media archiving platform for financial services and the public sector entities. They run a range of public archives, two of which include; the UK Government Web Archive and the UK Parliament Web Archive. |
Internet Archive (provides Archive-it service) | United States | 1996 | Heritrix, Wayback, NutchWAX Archived 2015-06-26 at the Wayback Machine and other tools developed by the Internet Archive | 150 | | Internet Archive's Wayback Machine is the largest and oldest web archive in the world, dating back to 1996. Internet Archive also provide various web archiving services, including Archive-IT, Save Page Now, and domain level contract crawls. The Wayback Machine is the publicly available access service to Internet Archive and partners' collections. |
Stanford University Libraries | United States | 2007 | Heritrix, HTTrack, Wayback, CDL Web Archiving Service, Internet Archive Archive-It | 2 | 5 | Stanford University Libraries has been engaged in web archiving projects since 2007 and started establishing a web archiving program in 2013. Collections that SUL is engaged in include Stanford University Archives, Bay Area Governments, Congressional Research Service (CRS) Reports, Freedom of Information Act (FOIA), Fugitive US Executive Agencies and many more. SUL is also involved in collaborative web archiving projects like the Archive of the California Government Domain, CA.gov with libraries at the University of California and the CA State Library, the End of Term Web Archive, and the Ivy Plus Libraries Confederation. |
Columbia University Libraries | United States | 2009 | Archive-it service | 2 | >1 | The Columbia University Libraries (CUL) web resources collection program archives selected websites in thematic areas corresponding to existing CUL collection strengths, websites produced by affiliates of Columbia University, and websites from organizations or individuals whose papers or records are held in CUL's physical archives. Began web archiving in 2008. |
Cornell University Library | United States | 2011 | Archive-it service | 1 | >1 | |
North Carolina State Government Web Site Archives | United States | 2005 | Archive-it service | | 3 | |
Latin American Web Archiving Project | United States | 2005 | Archive-it service | | | |
Web Archiving Project for the Pacific Islands | United States | 2009 | Archive-it service | | 4 | |
Library of Congress Web Archives | United States | 2000 | Heritrix, Wayback, and the DigiBoard, an in-house curatorial/permissions tool | 6 | 80 | The part time workers spend a few hours per month (on average) selecting content for the collections. |
Harvard Library | United States | 2006 | Archive-It | | >10 | Harvard Library web collections consist of 10 curatorial units' collections, with variable staff contributing to both technical and curatorial activities. Harvard is also involved in collaborative web collecting through the Ivy Plus Libraries Confederation. Harvard Library initiated web archiving activities in 2006 using a self-developed Web Archive Collection Service (WAX) and transitioned to Archive-It in 2017. |
Web Archiving Service from California Digital Library (WAS service) | United States | 2005 | Heritrix, Wayback, NutchWAX Archived 2015-06-26 at the Wayback Machine | 4 | >1 | The number of hours that curators devote to the service is very variable. |
Bentley Historical Library (University of Michigan) Web Archives | United States | 2000 | HTTrack, Teleport Pro, WAS service (2010-) | | 2 | |
University of Texas at San Antonio Web Archives | United States | 2009 | Archive-It | | 3 | The number of hours varies dependent upon how the crawls are scheduled. |
qumram | Switzerland | 2010 | qumram Web Archiving / Web Information Governance Software Suite | | | Commercial web archiving / web information governance software suite. Provides both remote harvesting as well as transactional web archiving. Allows integrations with any possible web application (WCMS, Portal, Sharepoint, eShop, custom applications) as well as repository (database, file system, electronic archive or records management system, cloud-based solution). Allows capturing and reproduction of public information as well as specific user interactions. |
SAPERION | Germany | 2011 | SAPERION ECM Web Content Archive | | | Commercial enterprise content management suite specializes on regulatory compliance. The product provides both harvesting as well as transactional web archiving based on the integration of qumram's Chronos Web Archiving Software Suite. Web content is just another channel from which content is reaching SAPERION. Others may be scanner, fax, e-mail, mobile devices, office suites or any other system creating content like ERP systems. |
Bibliotheca Alexandrina's Internet Archive | Egypt | 2002 | Heritrix, OpenWayback, WARCrefs | 3 | | Current crawling interests: Egypt beyond January 25, Arab League ccTLDs Deduplication: using WARCrefs tool to deduplicate Web archive content in BA clusterOpenWayback: handling big data indexing by using ZipNumCluster to locate a certain URI in compressed CDX files |
AUEB Web Archive | Greece | 2010 | Heritrix, Wayback and NutchWAX Archived 2015-06-26 at the Wayback Machine. | 1 | 1 | This project is part of the function of the University Library. |
World Bank Web Archives | United States | 2007 | HTTrack crawler, Oracle RDBMS, Google Search Appliance | 0 | 3 | |
Russian National Digital Archive | Russia | 2010 | wpull, grab-site, HTTrack crawler, ad-hoc scripts developed for social media archiving. Experimenting: Heritrix, Wayback | | | About 5000 government websites collected (May 2018) using wpull and provided as archives for downloading. |
Archive Team | Worldwide | 2009 | wpull, ad hoc scripts | 1 | ~100 | Volunteer group. They partially archived GeoCities, Yahoo! Videos, Google Video and others. |
WikiTeam | Worldwide | 2011 | ad hoc scripts | 0 | 0 | Volunteers group. Over 20,000 wikis preserved. |
University of North Texas CyberCemetery | United States | 1997 | Heritrix, Wayback; formerly HTTrack | | 2 | The CyberCemetery is an archive of government websites that have ceased operation (usually websites of defunct government agencies and commissions that have issued a final report). This collection features a variety of topics indicative of the broad nature of government information. In particular, this collection features websites that cover topics supporting the university's curriculum and particular program strengths. |
archive.today | Worldwide | 2012 | Apache Accumulo, HDFS, Chromium, ad hoc scripts | 1 | 1 | Saves external links from community web-sites (wikis, forums, blogs, ...). Can save snapshots of Web 2.0 pages. |
Greek Web Archive Portal | Greece | 2022 | Heritrix, Wayback | 0 | 1 | The Greek Web Archive Portal is a service provided by the National Library of Greece (NLG). It allows users to navigate through the historical content of the Greek Web, a separate collection of web content that includes snapshots of all .gr domain sites from 1996 up to the present day, harvested by the Internet Archive. The service was developed in collaboration with the Internet Archive and provides search either by keyword or by URL, covering web pages as well as other types of files: images, audio files, videos and PDFs. . |
ΕΣΑΕΙ Web Archive – National Archiving System of Greek Web | Greece | 2017 | Heritrix, Open Wayback, Solr, Netarchive Suite | 0 | 4 | The ΕΣΑΕΙ project was the first attempt to harvest all .gr content and get to know its dimensions. It was implemented by the National Library of Greece in collaboration with the Athens University of Economics and Business and it included two bulk and three selective harvests, regarding the collections of "Local Government", "News" and "Education“. NLG Curator Tool was created for the playback of the collection. |
Tamiment Library and Robert F. Wagner Labor Archives at New York University | United States | 2007 | WAS Service | 1 | 1 | Archives websites related to New York City and National Labor and Left Movements. Projects include: Alternative Mass Media / News; Anarchism; Animal Rights; Arts and Cultural Left; Civil Rights and Civil Liberties; Communism, Socialism, Trotskyism; Economic and Social Justice (Including Occupy Wall Street); Education and Student Movements; Electoral Politics and Parties / Political Action (U.S. Left); Environmentalism / Green Movement; Feminism and Women's Movements; Guantanamo Bay Detention Camp & War Crimes (U.S.); Housing; Internet/Cyberspace Democracy; Jewish American Progressive & Left Activity; Labor Unions and Organizations (U.S.); Left Academia and Theory, Intellectuals and Other Notables; LGBT Rights; Other Left Activism; Peace Movements; Prisoners Rights and Political Prisoners; Progressive Policy/ Educational Organizations. |
Preservica | Worldwide | 2012 | Heritrix, Preservica core product, Wayback | | | Cloud-based heterogeneous archiving service that allows ingest from multiple sources (including web archiving ingest via Heritrix). Ability to migrate content within WARC files and render in Wayback. Ingest runs as workflow so very little effort needed to run it. Developed, supported and run by Preservica. |
Central State Electronic Archives of Ukraine | Ukraine | 2007 | HTTrack, Wget | 2 | | Archives interested in keeping websites and creating the thematic collections of such websites, Is presently in storage the Archives collections of websites which includes the topic of presidential elections in Ukraine from 2010 until today, about the Chornobyl disaster, the local elections, of Euro 2012 in Ukraine, UNESCO World Heritage sites in Ukraine, the 200th anniversary of the birth of Taras Shevchenko. |
York University Libraries, York University Libraries Wayback Machine | Canada | 2012 | Browsertrix, pywb | 1 | 0 | |
New York Art Resources Consortium (NYARC) | United States | 2012 | Archive-It service | 1 | ~3 | Collaboration among Frick Art Reference Library, Brooklyn Museum Library & Archives, and Museum of Modern Art (MoMA) Library to archive specialist art historical web resources. |
Netherlands Institute for Sound and Vision (Sound and Vision) web archive | Netherlands | 2011 | Heritrix, Elasticsearch for full-text index, Drupal for front-end | | ~7 | Sound and Vision has been involved in web archiving projects since 2008, starting with the EU research project LiWA. After a couple of pilots, web archiving projects were scaled up in 2014. |
Rhizome (organization) | United States | 1999 | ArtBase, Webrecorder, Oldweb.Today | 3 | 1 | Rhizome operates a digital preservation program, led by Dragan Espenschied, which is focused on the creation of free, open source software tools to decentralize web archiving and software preservation practices and ensure access to its collections of born-digital art. Oldweb.Today and Webrecorder are its tools focused on web archiving specifically. |
University of Texas at Austin Libraries, Human Rights Documentation Initiative | United States | 2009 | Archive-It service | 1 | 1 | The University of Texas Libraries' Human Rights Documentation Initiative (HRDI) captures the websites of human rights organizations in order to provide secure access to human rights documentation in the event that these often-fragile sites are taken down. |
Kentucky Department for Libraries and Archives | United States | 2009 | Archive-it, Wayback | >1 | 0 | This collection includes captures of websites for Kentucky state agencies in the Executive, Legislative, and Judicial Branches. Stand-alone websites for boards, councils, committees, quasi-governmental agencies, and agency programs are also archived. Captures for websites dating 2000–2008 are included in this collection via a transfer to our account from the Wayback Machine. |
University of California, San Francisco Library | United States | 2007 | Archive-it, Wayback, CDL WAS Service | >1 | 0 | This collection documents the web presences of UCSF, as well as the larger health science focuses of AIDS history; anesthesiology; biotechnology and biomedical research; tobacco control and regulation; neuroscience; and computational medicine. Staff is one full-time digital archivist with various responsibilities in addition to web-archives. |
Ivy Plus Libraries Confederation | United States | 2013 | Archive-It, Conifer | 1 | 1 | The Ivy Plus Libraries Confederation's Web Resources Collection Program is a collaborative collection development effort to build curated, thematic collections of freely available, but at-risk, web content in order to support research at participating Libraries and beyond. Participating Libraries are: Brown, Chicago, Columbia, Cornell, Dartmouth, Duke, Harvard, Johns Hopkins, MIT, Penn, Princeton, Stanford, and Yale. Collections are accessible via Archive-It. |
Malaysian Government Web Archive (MyGWA) | Malaysia | 2017 | Wayback, WGET, WPULL | >1 | 0 | National Archive of Malaysia started to archive websites of public sector in Malaysia since 2017. |
HTTP Archive | | | | | | Crawls popular websites for Data analysis |
National Library of Medicine (U.S.) | United States | 2009 | Archive-It, Conifer | | ~8 | NLM web collecting is guided by the Collection Development Guidelines of the National Library of Medicine and other strategic collecting efforts. Collections include Global Health Events, the Opioid Epidemic, HIV/AIDS, Health and Medicine Blogs, and NLM's own web presence. |
Smithsonian Libraries and Archives (U.S.) | United States | 2000 | Heritrix, Archive-It, Webrecorder, Conifer, Browsertrix, other | | 5 | The Smithsonian Libraries and Archives collects websites and social media accounts that document the history of the Institution. |
Ghost Archive | United States | 2021 | Webrecorder | 1 | | |
Common Crawl | United States | 2008 | Apache Nutch, Apache Tika, pywb, in-house tools | 3 | 3 | |