Fetch HTML from another website in JavaScript

Why a direct fetch fails

The same-origin policy lets your page send a request anywhere, but the browser only hands the response to your code when it carries an Access-Control-Allow-Origin header that matches your origin. Web pages are made to be viewed, not read by scripts on other sites, so they send none. mode: 'no-cors' is not an escape hatch: it gives you an opaque response with status 0 and an empty body, explained in no-cors and opaque responses.

// Default mode: the browser blocks reading the response
await fetch('https://github.blog/');
// TypeError: Failed to fetch
// Console: No 'Access-Control-Allow-Origin' header is present on the requested resource.

// mode: 'no-cors' does not help: the response is opaque
const opaque = await fetch('https://github.blog/', { mode: 'no-cors' });
opaque.status; // 0
await opaque.text(); // '' (always empty)

Fetch through the proxy and parse with DOMParser

The proxy makes the request from its own servers, where no browser policy applies, and relays the status, Content-Type and body with Access-Control-Allow-Origin set to your origin. DOMParser with text/html turns the text into a document you can query with querySelector; it executes no scripts and loads no subresources. Relative links and image URLs in the parsed document still point at paths on the fetched site, so resolve them against the page URL (or its base element) before you use them.

const PROXY = 'https://proxy.cors.dev/';

async function fetchPage(pageUrl) {
  const response = await fetch(PROXY + pageUrl, { credentials: 'omit' });
  if (!response.ok) {
    const code = response.headers.get('X-Cors-Error'); // null when the site itself answered
    throw new Error(`HTTP ${response.status}${code ? ` (${code})` : ''} for ${pageUrl}`);
  }

  const doc = new DOMParser().parseFromString(await response.text(), 'text/html');

  // Make relative URLs usable outside the page they came from
  const base = doc.querySelector('base[href]')?.getAttribute('href') ?? pageUrl;
  for (const element of doc.querySelectorAll('a[href], img[src], link[href]')) {
    const attribute = element.hasAttribute('href') ? 'href' : 'src';
    try {
      element.setAttribute(attribute, new URL(element.getAttribute(attribute), base).href);
    } catch {
      element.removeAttribute(attribute);
    }
  }
  return doc;
}

const doc = await fetchPage('https://github.blog/');
const headings = [...doc.querySelectorAll('h2, h3')].map((heading) => heading.textContent.trim());
const links = [...doc.querySelectorAll('a[href^="https://github.blog/"]')].map((link) => link.href);
console.log(doc.title, headings.slice(0, 5), new Set(links).size);

Checked while writing this page, https://github.blog/ returned 285 KB of HTML through the proxy, https://unity.com/ 527 KB and https://www.theverge.com/ 971 KB, just under the free limit. Compressed transfer is handled by the proxy; the 1 MiB cap applies to the decoded body.

Extract text, links or data

Once parsed, extraction is ordinary DOM code: textContent for readable text, attribute reads for links and images, querySelectorAll with the selectors you would use in DevTools. JSON-LD blocks in script[type="application/ld+json"] are often the cleanest source for prices, events and articles, and JSON.parse on their text costs nothing.

function pageText(doc) {
  const copy = doc.body.cloneNode(true);
  copy.querySelectorAll('script, style, noscript, template, svg').forEach((node) => node.remove());
  return copy.textContent.replace(/\s+/g, ' ').trim();
}

const text = pageText(doc);
console.log(text.length, text.slice(0, 200));

What you get is the server-rendered page

• No JavaScript runs: a single-page app returns its empty shell. Look for the JSON API the app itself calls, which is usually easier to consume anyway.
• No cookies travel: you see the logged-out, default-region version of the page.
• Status codes pass through: a 404 or a 403 from the site arrives as that status with X-Cors-Source: upstream, while proxy-side problems set X-Cors-Error (response_too_large, upstream_timeout, owner_opted_out).
• HTTPS only: the proxy rejects http:// targets with invalid_request, and follows redirects for up to 4 hops as long as every hop is a public HTTPS host.
• Text only on the free tier: HTML, JSON, XML, CSV, plain text and SVG. Images and other binary types need Pro, which relays any content type up to 6 MiB.

Limits and pricing

When to move this to a server

Fetching one page on demand, when a user pastes a URL or opens a dashboard, is a browser job. Crawling many pages, rendering client-side apps with a headless browser, keeping a history, or scraping on a schedule while nobody has the tab open is a server job, and the fair-use pool will slow a browser crawl down long before a server would. Pro raises the ceiling to 600 requests per minute and 6 MiB pages for $5 per month, but it does not change where the code runs.

Good questions.