Wayback Machine API from JavaScript

Which Wayback Machine endpoints need a proxy

The Internet Archive splits its APIs across two hosts with different CORS rules. archive.org answers with Access-Control-Allow-Origin: *. web.archive.org, home of the CDX index and every capture, sends no CORS header, so the browser receives the response and refuses to hand it to your script. Prefixing those URLs with https://proxy.cors.dev/ fetches them from a server and returns them with the header. Reading needs no key anywhere.

Check if a URL is archived

https://archive.org/wayback/available?url=example.com returns the capture closest to now in about 200 B; add &timestamp=20060101 for the one closest to a date. archived_snapshots.closest holds available, url, timestamp and status, and archived_snapshots is empty when the URL was never archived.

// Markup: <p id="capture"></p>
// No proxy: archive.org/wayback/available sends Access-Control-Allow-Origin: *
async function closestCapture(url, timestamp) {
  const query = new URLSearchParams({ url });
  if (timestamp) query.set('timestamp', timestamp);
  const response = await fetch('https://archive.org/wayback/available?' + query, {
    credentials: 'omit',
  });
  if (!response.ok) throw new Error(`Availability check failed: ${response.status}`);
  const { archived_snapshots: snapshots } = await response.json();
  if (!snapshots.closest) return null;
  return {
    timestamp: snapshots.closest.timestamp,
    url: snapshots.closest.url.replace(/^http:/, 'https:'),
  };
}

const output = document.querySelector('#capture');
closestCapture('example.com', '20060101')
  .then((capture) => {
    output.textContent = capture ? `${capture.timestamp}: ${capture.url}` : 'Not archived';
  })
  .catch((error) => {
    output.textContent = error.message;
  });

The url field starts with http:// even for https sites, so the example switches it before you link to it. On web.archive.org the same path is a 404. archive.org sends Cache-Control: max-age=21600, so the browser reuses an answer for up to 6 hours.

Wayback CDX API: every capture of a URL

The CDX server at https://web.archive.org/cdx/search/cdx lists captures, one row each. With output=json, the first row holds the column names: urlkey, timestamp, original, mimetype, statuscode, digest and length. It sends no CORS header, so the example prefixes it.

// Top-level await: run it in a module script or the DevTools console
const CDX = 'https://proxy.cors.dev/https://web.archive.org/cdx/search/cdx';

async function captures(url, options = {}) {
  const query = new URLSearchParams({ url, output: 'json', limit: '100', ...options });
  const response = await fetch(`${CDX}?${query}`, { credentials: 'omit' });
  if (!response.ok) {
    throw new Error(`CDX request failed: ${response.headers.get('X-Cors-Error') ?? response.status}`);
  }
  const [fields, ...rows] = await response.json();
  if (!fields) return [];
  return rows
    .filter((row) => row.length === fields.length)
    .map((row) => Object.fromEntries(fields.map((field, index) => [field, row[index]])));
}

const yearly = await captures('example.com', {
  fl: 'timestamp,original,statuscode',
  filter: 'statuscode:200',
  collapse: 'timestamp:4',
  from: '2015',
  to: '2020',
});
console.table(yearly);

• url: the page to look up; matchType=prefix, host or domain, or a trailing /*, widens it to a path or host.
• from and to: timestamps such as 2015 or 20150301.
• fl: the columns to return, such as timestamp,original,statuscode.
• filter: field:regex, repeatable; ! before the field excludes matches, as in filter=!statuscode:200.
• collapse: drops adjacent rows with the same value. collapse=digest skips captures whose content did not change and collapse=timestamp:4 keeps one per year.
• limit: maximum rows; limit=-1 returns only the latest capture (201 B). Always set it: an unlimited query for example.com passes 1 MiB and the proxy answers 502 response_too_large.
• showResumeKey=true: appends a resume key; pass it back as resumeKey for the next batch.

Wide queries are slow. url=paulgraham.com/* took 37 seconds directly, and through the proxy prefix and domain queries for example.com ended in 503 or a 504 upstream_timeout after the free tier's 10 seconds. Even a single-URL query timed out once in two tries, so narrow with from, to and limit and retry a 504 once.

Fetch archived HTML with id_

A capture URL is https://web.archive.org/web/<timestamp>/<url>. Requested as is, it redirects to the nearest capture and returns the replay version, with links rewritten to web.archive.org and Wayback's bundle-playback.js and wombat.js scripts added: 2.9 KB for example.com instead of 1.3 KB. Add id_ right after the timestamp to get the page exactly as captured.

// Markup: <p id="archived"></p>
async function archivedPage(timestamp, url) {
  const snapshot = `https://web.archive.org/web/${timestamp}id_/${url}`;
  const response = await fetch('https://proxy.cors.dev/' + snapshot, { credentials: 'omit' });
  if (!response.ok) {
    throw new Error(`Snapshot request failed: ${response.headers.get('X-Cors-Error') ?? response.status}`);
  }
  return new DOMParser().parseFromString(await response.text(), 'text/html');
}

const output = document.querySelector('#archived');
archivedPage('20200101000000', 'https://example.com/')
  .then((archived) => {
    const links = [...archived.querySelectorAll('a[href]')].map((link) => link.getAttribute('href'));
    output.textContent = `${archived.title}: ${links.length} links, first ${links[0] ?? 'none'}`;
  })
  .catch((error) => {
    output.textContent = error.message;
  });

Even a full timestamp redirects: 20200101000000id_ landed on 20191231234501, and a short 2020id_ on a capture from January 2021. The proxy follows up to 4 redirects but does not say which capture it reached, so take exact timestamps from CDX. Wikipedia's main page (74 KB) and the BBC home page (229 KB) worked; CNN's 1.1 MB page returned 502 response_too_large, and Pro raises the limit to 6 MiB. im_ returns image bytes, which the free tier refuses; an <img> tag needs no CORS, so point it at the capture URL instead.

Search and metadata: archive.org, no proxy

Search and metadata send Access-Control-Allow-Origin: *, so call them directly. https://archive.org/advancedsearch.php takes a query such as collection:nasa AND mediatype:movies in q, repeated fl[] fields, rows, page and output=json; results are in response.docs and the total in response.numFound. For long result sets, https://archive.org/services/search/v1/scrape returns items, total and a cursor for the next batch.

// No proxy: archive.org search and metadata send Access-Control-Allow-Origin: *
// Top-level await: run it in a module script or the DevTools console
async function archiveJson(url) {
  const response = await fetch(url, { credentials: 'omit' });
  if (!response.ok) throw new Error(`archive.org request failed: ${response.status}`);
  return response.json();
}

const query = new URLSearchParams({ q: 'collection:nasa AND mediatype:movies', rows: '5', output: 'json' });
query.append('fl[]', 'identifier');
query.append('fl[]', 'title');
const { response: results } = await archiveJson('https://archive.org/advancedsearch.php?' + query);
console.log(`${results.numFound} items`, results.docs);

const item = await archiveJson('https://archive.org/metadata/' + results.docs[0].identifier);
if (item.metadata) {
  console.log(item.metadata.title, `${item.files.length} files`);
}

https://archive.org/metadata/<identifier> returns one item: metadata such as title, date and collection, files with each name and format, and server and dir for direct file links. The nasa collection is 6.7 KB and a scanned book with 27 files 56 KB; add /metadata to the URL for only that block (1.9 KB). An unknown identifier returns {} with status 200.

Save Page Now belongs on a server

Save Page Now 2 is a POST to https://web.archive.org/save with the header Authorization: LOW accesskey:secret, using the S3-style keys from your archive.org account, followed by polling /save/status/<job_id>. The secret must stay on your server or Worker: anything in browser code is readable in DevTools, and cors.dev never stores or adds upstream keys. Run captures from your backend and read the results from the browser.

Rate limits and errors

• Internet Archive limits: bursts get 429. Internet Archive staff have described the CDX limit as about 60 requests per minute, with a one-hour firewall block, doubling on repeat, for clients that ignore 429s. Our test machine was refused connections for a few minutes after a burst.
• A throttled lookup looks like a CORS error: archive.org's 429 page has no Access-Control-Allow-Origin header, so fetch() rejects with a CORS error instead of returning 429. Through the proxy, a 429 arrives with CORS headers and your code can read it.
• Free tier rules: GET and HEAD only, up to 1 MiB, no daily or monthly quota, 600 requests per minute per client IP, 10 seconds for the archive to answer, and only Accept, If-None-Match and If-Modified-Since are forwarded. Pro adds POST, custom headers and 6 MiB responses for $5 per month.

Good questions.