Menu
Home Explore People Places Arts History Plants & Animals Science Life & Culture Technology
On this page
Web archiving
Preserving history of web pages

Web archiving involves collecting, preserving, and providing access to content from the World Wide Web to ensure information is preserved in an archival format for research and public use. Archivists use automated web crawlers to capture vast amounts of web data. Notable services like the Wayback Machine, operated by the Internet Archive, exemplify such efforts. As more cultural content emerges online, institutions such as national libraries and national archives increasingly engage in web archiving to prevent data loss. Additionally, commercial web archiving solutions help organizations preserve their own content for regulatory, legal, or heritage reasons.

We don't have any images related to Web archiving yet.
We don't have any YouTube videos related to Web archiving yet.
We don't have any PDF documents related to Web archiving yet.
We don't have any Books related to Web archiving yet.

History and development

While curation and organization of the web has been prevalent since the mid- to late-1990s, one of the first large-scale web archiving projects was the Internet Archive, a non-profit organization created by Brewster Kahle in 1996.3 The Internet Archive released its own search engine for viewing archived web content, the Wayback Machine, in 2001.4 As of 2018, the Internet Archive was home to 40 petabytes of data.5 The Internet Archive also developed many of its own tools for collecting and storing its data, including PetaBox for storing large amounts of data efficiently and safely, and Heritrix, a web crawler developed in conjunction with the Nordic national libraries.6 Other projects launched around the same time included a web archiving project by the National Library of Canada, Australia's Pandora, Tasmanian web archives and Sweden's Kulturarw3.78

From 2001 to 2010, the International Web Archiving Workshop (IWAW) provided a platform to share experiences and exchange ideas.910 The International Internet Preservation Consortium (IIPC), established in 2003, has facilitated international collaboration in developing standards and open source tools for the creation of web archives.11

The now-defunct Internet Memory Foundation was founded in 2004 and founded by the European Commission in order to archive the web in Europe.12 This project developed and released many open source tools, such as "rich media capturing, temporal coherence analysis, spam assessment, and terminology evolution detection."13 The data from the foundation is now housed by the Internet Archive, but not currently publicly accessible.14

Despite the fact that there is no centralized responsibility for its preservation, web content is rapidly becoming the official record. For example, in 2017, the United States Department of Justice affirmed that the government treats the President's tweets as official statements.15

Methods of collection

See also: List of Web archiving initiatives and List of web archiving file formats

Web archivists generally archive various types of web content including HTML web pages, style sheets, JavaScript, images, and video. They also archive metadata about the collected resources such as access time, MIME type, and content length. This metadata is useful in establishing authenticity and provenance of the archived collection.

Transactional archiving

Transactional archiving is an event-driven approach, which collects the actual transactions which take place between a web server and a web browser. It is primarily used as a means of preserving evidence of the content which was actually viewed on a particular website, on a given date. This may be particularly important for organizations which need to comply with legal or regulatory requirements for disclosing and retaining information.16

A transactional archiving system typically operates by intercepting every HTTP request to, and response from, the web server, filtering each response to eliminate duplicate content, and permanently storing the responses as bitstreams.

Difficulties and limitations

Crawlers

Web archives which rely on web crawling as their primary means of collecting the Web are influenced by the difficulties of web crawling:

  • The robots exclusion protocol may request crawlers not access portions of a website. Some web archivists may ignore the request and crawl those portions anyway.
  • Large portions of a website may be hidden in the Deep Web. For example, the results page behind a web form can lie in the Deep Web if crawlers cannot follow a link to the results page.
  • Crawler traps (e.g., calendars) may cause a crawler to download an infinite number of pages, so crawlers are usually configured to limit the number of dynamic pages they crawl.
  • Most of the archiving tools do not capture the page as it is. It is observed that ad banners and images are often missed while archiving.

However, it is important to note that a native format web archive, i.e., a fully browsable web archive, with working links, media, etc., is only really possible using crawler technology.

The Web is so large that crawling a significant portion of it takes a large number of technical resources. Also, the Web is changing so fast that portions of a website may suffer modifications before a crawler has even finished crawling it.

General limitations

Some web servers are configured to return different pages to web archiver requests than they would in response to regular browser requests. This is typically done to fool search engines into directing more user traffic to a website and is often done to avoid accountability or to provide enhanced content only to those browsers that can display it.

Not only must web archivists deal with the technical challenges of web archiving, they must also contend with intellectual property laws. Peter Lyman17 states that "although the Web is popularly regarded as a public domain resource, it is copyrighted; thus, archivists have no legal right to copy the Web". However national libraries in some countries18 have a legal right to copy portions of the web under an extension of a legal deposit.

Some private non-profit web archives that are made publicly accessible like WebCite, the Internet Archive or the Internet Memory Foundation allow content owners to hide or remove archived content that they do not want the public to have access to. Other web archives are only accessible from certain locations or have regulated usage. WebCite cites a recent lawsuit against Google's caching, which Google won.19

Laws

In 2017 the Financial Industry Regulatory Authority, Inc. (FINRA), a United States financial regulatory organization, released a notice stating all the businesses doing digital communications are required to keep a record. This includes website data, social media posts, and messages.20 Some copyright laws may inhibit Web archiving. For instance, academic archiving by Sci-Hub falls outside the bounds of contemporary copyright law. The site provides enduring access to academic works including those that do not have an open access license and thereby contributes to the archival of scientific research which may otherwise be lost.2122

See also

  • Internet portal

General bibliography

References

  1. "Web Archiving". Netpreserve - International Internet Preservation Consortium. August 14, 2024. Archived from the original on July 12, 2024. https://netpreserve.org/web-archiving/ ↩

  2. Truman, Gail (2016). "Web Archiving Environmental Scan". Harvard Library. https://nrs.harvard.edu/urn-3:HUL.InstRepos:25658314 ↩

  3. Toyoda, M.; Kitsuregawa, M. (May 2012). "The History of Web Archiving". Proceedings of the IEEE. 100 (Special Centennial Issue): 1441–1443. doi:10.1109/JPROC.2012.2189920. ISSN 0018-9219. https://doi.org/10.1109%2FJPROC.2012.2189920 ↩

  4. Toyoda, M.; Kitsuregawa, M. (May 2012). "The History of Web Archiving". Proceedings of the IEEE. 100 (Special Centennial Issue): 1441–1443. doi:10.1109/JPROC.2012.2189920. ISSN 0018-9219. https://doi.org/10.1109%2FJPROC.2012.2189920 ↩

  5. "Inside Wayback Machine, the internet's time capsule". The Hustle. September 28, 2018. sec. Wayyyy back. Retrieved July 21, 2020. https://thehustle.co/inside-wayback-machine-internet-archive ↩

  6. Toyoda, M.; Kitsuregawa, M. (May 2012). "The History of Web Archiving". Proceedings of the IEEE. 100 (Special Centennial Issue): 1441–1443. doi:10.1109/JPROC.2012.2189920. ISSN 0018-9219. https://doi.org/10.1109%2FJPROC.2012.2189920 ↩

  7. Costa, Miguel; Gomes, Daniel; Silva, Mário J. (September 2017). "The evolution of web archiving". International Journal on Digital Libraries. 18 (3): 191–205. doi:10.1007/s00799-016-0171-9. S2CID 24303455. /wiki/Doi_(identifier) ↩

  8. Consalvo, Mia; Ess, Charles, eds. (April 2011). "Web Archiving – Between Past, Present, and Future". The Handbook of Internet Studies (1 ed.). Wiley. pp. 24–42. doi:10.1002/9781444314861. ISBN 978-1-4051-8588-2. 978-1-4051-8588-2 ↩

  9. "IWAW 2010: The 10th Intl Web Archiving Workshop". www.wikicfp.com. Retrieved August 19, 2019. https://www.wikicfp.com/cfp/servlet/event.showcfp?eventid=9651 ↩

  10. "IWAW - International Web Archiving Workshops". bibnum.bnf.fr. Archived from the original on November 20, 2012. Retrieved August 19, 2019. https://web.archive.org/web/20121120035514/https://bibnum.bnf.fr/ecdl/index.html ↩

  11. "About the IIPC". IIPC. Retrieved April 17, 2022. https://netpreserve.org/about-us/ ↩

  12. Toyoda, M.; Kitsuregawa, M. (May 2012). "The History of Web Archiving". Proceedings of the IEEE. 100 (Special Centennial Issue): 1441–1443. doi:10.1109/JPROC.2012.2189920. ISSN 0018-9219. https://doi.org/10.1109%2FJPROC.2012.2189920 ↩

  13. Toyoda, M.; Kitsuregawa, M. (May 2012). "The History of Web Archiving". Proceedings of the IEEE. 100 (Special Centennial Issue): 1441–1443. doi:10.1109/JPROC.2012.2189920. ISSN 0018-9219. https://doi.org/10.1109%2FJPROC.2012.2189920 ↩

  14. "Internet Memory Foundation : Free Web: Free Download, Borrow and Streaming". archive.org. Internet Archive. Retrieved July 21, 2020. https://archive.org/details/internetmemoryfoundation ↩

  15. Regis, Camille (June 4, 2019). "Web Archiving: Think the Web is Permanent? Think Again". History Associates. Retrieved July 14, 2019. https://www.historyassociates.com/resources/blog/web-archiving-challenges/ ↩

  16. Brown, Adrian (January 10, 2016). Archiving websites: a practical guide for information management professionals. Facet. ISBN 978-1-78330-053-2. OCLC 1064574312. 978-1-78330-053-2 ↩

  17. Lyman (2002) ↩

  18. "Legal Deposit | IIPC". netpreserve.org. Archived from the original on March 16, 2017. Retrieved January 31, 2017. https://netpreserve.org/legal-deposit ↩

  19. "WebCite FAQ". Webcitation.org. Retrieved September 20, 2018. https://www.webcitation.org/faq ↩

  20. "Social Media and Digital Communications" (PDF). finra.org. FINRA. https://www.finra.org/sites/default/files/notice_doc_file_ref/Regulatory-Notice-17-18.pdf ↩

  21. Claburn, Thomas (September 10, 2020). "Open access journals are vanishing from the web, Internet Archive stands ready to fill in the gaps". The Register. https://www.theregister.com/2020/09/10/open_access_journal/ ↩

  22. Laakso, Mikael; Matthias, Lisa; Jahn, Najko (2021). "Open is not forever: A study of vanished open access journals". Journal of the Association for Information Science and Technology. 72 (9): 1099–1112. arXiv:2008.11933. doi:10.1002/ASI.24460. S2CID 221340749. /wiki/ArXiv_(identifier) ↩