WEB ARCHIVING · CRAWLING · WARC · REPLAY · LINK ROT · LEGAL DEPOSIT · DIGITAL HERITAGE

How Web Archiving Works

The web feels permanent because pages can be copied instantly. In reality, websites are among the most unstable publication forms ever created.

A website can disappear without burning, decaying or being physically destroyed. Someone only has to stop serving it.

Domains expire. Content-management systems are replaced. Pages are redesigned. Images move. APIs shut down. JavaScript changes. Social platforms remove posts. Organisations close. A page that was public yesterday can return an error today while search engines slowly forget it ever existed.

Web archiving responds by capturing selected states of websites and preserving enough files, metadata and relationships for future users to reconstruct what was publicly available at particular times.

The short answer

SELECT TARGET
  → DEFINE SCOPE
  → CRAWL / CAPTURE
  → FETCH PAGE + DEPENDENCIES
  → RECORD URL / TIME / RESPONSE
  → PACKAGE CAPTURE
  → QUALITY CHECK
  → STORE REDUNDANTLY
  → INDEX
  → REPLAY
  → RE-CRAWL LATER
  → PRESERVE CHANGE OVER TIME

1. A website is not a single file

A modern web page may depend on HTML, CSS, JavaScript, images, fonts, video, APIs, databases, cookies and third-party services. Capturing only the visible text may preserve information but lose design, interaction and context.

Web archiving therefore attempts to preserve a networked publication state rather than a single document.

2. Crawlers follow links

A web crawler begins from one or more seed URLs, requests pages and follows links according to configured rules.

The crawler may remain inside one domain, follow selected external resources, limit depth or stop after a defined number of files. Scope determines what becomes part of the capture.

3. Scope is an archival decision

Should the archive capture every page on a site? External PDFs? Embedded YouTube videos? Social feeds? Search results? Logged-in areas? Comment threads?

There is no universal answer. The archive defines a collection policy based on mission, law, technical feasibility, significance and resources.

4. A crawl is a snapshot, not the whole website

Websites can change while the crawler is running. One page may be captured at 10:00 and another at 10:20 after an update. A large crawl therefore approximates a state rather than freezing the site at one exact instant.

Capture timestamps are essential because the archive needs to know when each resource was retrieved.

5. HTTP responses are part of the evidence

A web capture can preserve the requested URL, response status, headers, content type and timestamp alongside the payload.

This helps future systems distinguish successful captures, redirects, errors and missing resources.

6. WARC packages web captures

The WARC format, standardised internationally, is widely used for web archiving. It can package web resources and associated metadata into archival container files.

WARC does not magically make a website permanent, but it gives captured network transactions a preservation-friendly structure.

7. Replay reconstructs the archived experience

Replay software reads archived captures and rewrites links so a user can navigate preserved pages rather than the live web.

The result can feel like visiting the original site, but it is a reconstruction based on what the crawl successfully captured.

8. Missing resources create broken replay

If an image, stylesheet, script or external service was not captured, the archived page may look incomplete or behave differently.

A quality-assurance process should identify major capture failures while accepting that perfect reproduction is often impossible.

9. JavaScript changed web archiving

Older websites sent most content in initial HTML. Modern sites may construct pages after loading by executing JavaScript and querying APIs.

Traditional crawlers can miss this content. Browser-based capture tools and more sophisticated crawling strategies may be needed to preserve dynamic experiences.

10. APIs are hidden dependencies

A page can render today because it calls a live API. If the API disappears, the archived HTML may no longer show the information.

Preserving dynamic sites therefore requires understanding where visible content actually comes from.

11. Video is difficult to capture comprehensively

Streaming media may use segmented delivery, authentication, adaptive bitrate and third-party platforms. Capturing the surrounding page does not necessarily preserve the video.

Archives may need separate audiovisual acquisition workflows for material judged significant.

12. Social media is not simply another website

Feeds are personalised, algorithmic and account-dependent. Comments and engagement counts change. APIs impose limits. Platforms control access.

A capture may preserve a public page or selected posts without reproducing the personalised experience that individual users saw.

13. Link rot erodes public knowledge

Link rot occurs when a URL no longer resolves to the resource it once identified. The page may vanish, move or be replaced.

Web archives give citations a possible historical return path even after the live source disappears.

14. Content drift is different from link rot

A URL may still work while the content has changed substantially. This creates content drift: the citation looks alive, but the evidence a writer originally saw is gone.

Timestamped web captures help reconstruct which version existed at the relevant date.

15. Repeated crawls create temporal depth

One capture documents one state. Repeated captures create a time series.

Researchers can then examine changes in policy, language, design, institutional structure, prices, public messaging and cultural priorities.

16. Crawl frequency should match volatility

A static heritage website may need infrequent capture. An election site, crisis-response page or breaking-news publication may change many times per day.

Web archiving is therefore a scheduling problem as well as a collection problem.

17. Event harvesting captures short-lived public moments

Archives sometimes create rapid collection programmes around elections, disasters, cultural events or major public transitions.

The challenge is to identify important web sources before they change or disappear.

18. Domain harvesting captures a broad national web

National web archives may conduct broad crawls across a country-code domain. This sacrifices some capture depth in exchange for wide coverage.

Selective collections can then add deeper or more frequent captures for high-value sites.

19. Singapore’s Web Archive began before digital legal deposit

The National Library of Singapore began web-archiving activity in 2006. Web Archive Singapore preserves Singapore-related websites and makes selected captures discoverable.

This shows how archival practice can precede legislation and later become integrated into a stronger national preservation framework.

20. Singapore expanded legal deposit to web archiving in 2018

Singapore amended the National Library Board Act in 2018 to expand legal-deposit coverage to digital materials and support archiving of websites containing the .sg domain without requiring individual written permission for each capture within the statutory framework.

This was a major shift from object deposit toward network capture as national-memory infrastructure.

21. Legal authority and technical capability are separate

A legal mandate may permit capture, but it does not guarantee that every modern website can be preserved technically.

Authentication, encryption, streaming, robots directives, complex scripts and external services can still limit what an archive actually captures.

22. Robots rules have archival complexity

Robots exclusion conventions were designed primarily to communicate crawling preferences to automated agents. Archival institutions may have collection policies or legal mandates that interact with those conventions differently from commercial search engines.

The archive should document its policy rather than pretending that every crawler operates under the same purpose and authority.

23. Access can be more restricted than capture

An institution may be authorised to preserve content but still limit how archived copies are made available because of copyright, privacy, licensing or other legal considerations.

Preservation and public redistribution remain separate decisions.

24. Personal data does not disappear when the live page disappears

An archived website may preserve personal information that the live publisher later removes. This creates ethical and legal questions around continuing access.

Archives need policies for sensitive information, takedown requests where applicable, restricted access and lawful preservation.

25. Search indexing creates a new representation

Archived captures become much more useful when full-text indexed. But index text may derive from imperfect parsing or OCR.

The search layer should always return users to the archived capture rather than acting as the sole evidentiary object.

26. Replay dates must be visible

A user should be able to distinguish a 2017 capture from a 2026 capture. Without a visible timestamp, archived web content can be mistaken for current information.

This is especially important for laws, health advice, prices, curricula and other time-sensitive information.

27. The archive should preserve capture provenance

Useful metadata can include seed URL, crawl date, capture time, software, collection, scope, response status and checksums.

These records help future users understand how the archived page came into existence.

28. Web archives need digital preservation after capture

WARC files can corrupt. Storage fails. Index formats age. Replay software changes.

Web archiving therefore includes the same fixity, redundancy, format monitoring and preservation planning required by other digital archives.

29. Captures can be very large

Broad crawls generate huge datasets. Storage cost is only part of the problem: indexing, replication, integrity checking, migration and access infrastructure also scale.

Collection policy is therefore a resource-allocation system.

30. Deduplication reduces repeated storage

Repeated crawls often encounter unchanged files. Preservation systems can use deduplication or revisit records to avoid storing identical payloads repeatedly while preserving crawl history.

The design must preserve enough evidence to reconstruct what the crawler saw at each time.

31. Web archives are not perfect mirrors

An archived site may be incomplete, technically distorted or missing embedded services. Researchers should treat archived websites as preserved captures with documented limitations.

This is analogous to using a damaged historical photograph: valuable evidence, but not a magical reconstruction of the entire moment.

32. The missing page can itself be evidence

If a capture repeatedly returns a 404 or removal notice, that state may document a historical transition.

Web archiving preserves absence and failure states as well as successful pages.

33. Publication corrections make web archives more important

A publisher can silently rewrite a web page. Archived snapshots help establish what earlier readers actually saw.

This makes web archives important for accountability, citation and edition history.

34. Web archiving supports research across disciplines

Historians can examine public communication. Linguists can study changing language. Political scientists can compare campaign messages. Designers can study interface history. Economists can examine old product and price information. Sociologists can study institutional change.

The web is not merely technical infrastructure. It is a cultural record.

35. AI can search web archives at scale

AI can extract entities, cluster pages, identify changes, summarise capture sets and compare language across time.

The system should preserve URL, capture timestamp and archived-object citation so the synthesis remains verifiable.

QUESTION
  → ARCHIVED URLS
  → CAPTURE DATES
  → CONTENT EXTRACTION
  → CHANGE DETECTION
  → SYNTHESIS
  → CITED CAPTURE
  → REPLAY
  → USER VERIFIES HISTORICAL STATE

36. AI should not collapse time

If an AI reads ten captures from ten years and synthesises them without dates, it can create a fictional timeless website that never existed.

Temporal provenance is therefore mandatory for serious web-archive analysis.

37. Failure modes

Failure	What breaks
Save HTML only	Images, style and dependencies may vanish.
No capture timestamp	Archived state cannot be placed in time.
Assume crawl is complete	Missing dynamic resources are overlooked.
Capture once	Change through time disappears.
Replay without date banner	Historical content may be mistaken for current guidance.
Index treated as source	Extracted text replaces preserved capture as evidence.
No digital preservation	WARC files become another fragile file collection.
AI synthesis without timestamps	Multiple historical states collapse into one false present.

38. A practical web-archiving checklist

Define collection purpose.
Select seed URLs.
Set crawl scope and frequency.
Capture dependencies where feasible.
Preserve URL, timestamp and response metadata.
Use archival packaging such as WARC where appropriate.
Quality-check major capture failures.
Store captures redundantly with fixity.
Index for discovery without losing provenance.
Make replay dates obvious.
Re-crawl volatile sites.
Document legal and access policies.
Preserve record-level citations for AI and research use.

39. The deeper model: web archiving preserves change, not just pages

A printed book usually reaches publication as a defined edition. A website keeps moving.

The archive therefore preserves a sequence of observable states:

SITE AT T1
  → SITE AT T2
  → SITE AT T3
  → SITE AT T4
  → DIFFERENCES BECOME HISTORY

That sequence is what makes the historical web researchable.

The web remembers by repetition: capture, return, capture again.

Source and authority routes

Continue the Archives and Publishing series

Publication control: Wintour House · eduKate Publishing · evidence, capture, preservation, edition and archive gates.

World Return: When a website matters, do not assume the live URL will remain the evidence. Capture its state, preserve the timestamp, keep enough dependencies to understand it, and return later so change itself becomes part of the archive.

How Web Archiving Works | From Crawling and Capture to Replay, Legal Deposit, Preservation and the Historical Web