Internet Is Fun 
A surprising amount of the internet’s memory rests on a small number of organisations and a rather larger number of volunteers. It works well enough that most people assume it is automatic and comprehensive. It is neither.
Understanding roughly how it works is useful, partly because it tells you what to do when you find something worth keeping.
The basic mechanism
An archive crawler is a program that fetches a page, stores the response exactly as it arrived β headers, HTML, images, stylesheets β and then follows the links to do the same again. The result is written into a standard container format designed for this purpose, which records not just the content but the circumstances of the fetch.
Playing it back means reconstructing the page from those stored responses, rewriting links so they point at other archived copies rather than the live web. When you look at a page from 2004, you are looking at a reassembly, not a photograph.
Who is doing it
The Internet Archive is the large one. Its Wayback Machine has been capturing pages since 1996 and holds an enormous number of snapshots. It crawls continuously and also accepts pages submitted by hand.
National libraries run web archives as part of legal deposit β preserving the portion of the web belonging to their country. These are often more thorough than the Internet Archive within their remit and much less visible outside it.
Archive Team is a volunteer group that mobilises when a service announces it is shutting down. Their rescue of GeoCities before the 2009 shutdown is the best-known example, and it is the reason a great deal of that material still exists. More on that here.
On-demand services take a single snapshot of one page when someone asks, which is a different job from crawling and is better at some pages the big crawlers struggle with.
Why pages fail to archive
This is the part most people have not thought about, and it explains a lot of frustrating gaps.
- The content is built by JavaScript. If the page arrives empty and fills itself in by calling an API, the crawler may store an empty page. This is the single biggest cause of blank archived pages.
- Infinite scroll. There are no links to follow, so the crawler sees the first screen and stops.
- Logins and paywalls. Crawlers are not signed in, so they get what a stranger gets.
- Media on separate servers. Video and images served from a content network often are not captured with the page, so the text survives and the pictures do not.
- Nobody ever visited. Crawlers prioritise what is linked to. A page nothing links to may simply never be reached.
The pattern: the older and simpler the page, the better it archives. A 1998 hobby page is preserved almost perfectly. A 2023 app-like site may be a blank rectangle.
How to save something yourself
If you find something you would be upset to lose, do not assume it is covered.
- Submit it. The Wayback Machine has a “Save Page Now” function that captures a page immediately. It takes about ten seconds and it is the single most useful thing an individual can do.
- Keep your own copy. Print to PDF, or save the complete page. An archive you control does not depend on anyone else’s funding.
- Save the address as well as the content. A file with no record of where it came from loses most of its value as evidence.
Why this is worth caring about
Web archives are the only reason it is possible to check what a page said last year, whether a site existed before last month, or what an argument actually was before it was summarised. That makes them infrastructure for verification, not just nostalgia β see how to check if a website is safe for the practical version.
They also run on a fraction of the money you would assume, largely on donations and goodwill. For what happens to the parts nobody caught in time, see lost media.
