Scrape

Scrape is the core primitive. Send a single URL and get it back the way a model wants to read it: clean Markdown, the page’s own structured data, extracted JSON, links, or a summary. You describe the formats and how the page should be fetched; Hydrafetch handles the rest.

When to use

You have a specific URL and want clean content from it right now.
You need more than Markdown — the page’s structured data, its links, or an LLM summary.
You want control over how the page is fetched: render JavaScript-heavy pages, wait for content to settle, or fetch as if from another country.

For a whole site, use Crawl. For an explicit list of URLs, use Batch.

Example request

Send a POST to /v1/web/scrape with your key in the X-API-Key header. Omit formats for Markdown only.

curl -X POST https://api.hydrafetch.com/v1/web/scrape \
  -H "X-API-Key: hf_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "formats": ["markdown", "links", "structured"]
  }'

const res = await fetch("https://api.hydrafetch.com/v1/web/scrape", {
  method: "POST",
  headers: {
    "X-API-Key": "hf_your_key_here",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    url: "https://example.com",
    formats: ["markdown", "links", "structured"],
  }),
});
const { data } = await res.json();
console.log(data.markdown);

import requests

res = requests.post(
    "https://api.hydrafetch.com/v1/web/scrape",
    headers={"X-API-Key": "hf_your_key_here"},
    json={
        "url": "https://example.com",
        "formats": ["markdown", "links", "structured"],
    },
)
print(res.json()["data"]["markdown"])

Example response

The response is { "data": ... }. Only the formats you requested appear as fields; metadata and usage are always present.

{
  "data": {
    "url": "https://example.com",
    "finalUrl": "https://example.com/",
    "status": 200,
    "cached": false,
    "metadata": {
      "title": "Example Domain",
      "pageType": "article",
      "wordCount": 214
    },
    "usage": { "creditsUsed": 1, "creditsRemaining": 4999, "freshness": "fresh" },
    "markdown": "# Example Domain\n\nThis domain is for use in illustrative examples...",
    "links": {
      "internal": ["https://example.com/about"],
      "external": ["https://www.iana.org/domains/example"]
    },
    "structured": {
      "entities": [
        { "type": "Organization", "source": "json-ld", "properties": { "name": "Example" } }
      ],
      "jsonLd": [],
      "microdata": [],
      "opengraph": [],
      "rdfa": [],
      "appState": []
    }
  }
}

Request options

Content

string

required

The URL to scrape. Must be http(s).

string[]

default:"[\"markdown\"]"

What to return: markdown, html, rawHtml, links, structured, summary, json. structured is the page’s own data (cheapest, no LLM); summary and json are LLM-backed and cost more. When you request json, supply jsonOptions.

object

Schema and/or prompt for the json format. Provide jsonOptions.schema (a JSON Schema describing the shape you want) and/or jsonOptions.prompt (a natural-language instruction). One is required when json is requested.

boolean

default:"false"

Preserve document structure (headings, lists, tables) over prose density. Good for marketing and service pages, where the headings are the content.Leave it off and the extractor optimises for capturing every word, which scores better on articles but can return unstructured prose on listing-style pages. Check metadata.structure in the response: if it came back plain and you wanted markdown, retry with this on.

Content filters

boolean

default:"true"

Return only the main content, dropping nav and boilerplate.

string[]

CSS selectors to keep. When set, only matching elements are considered. Up to 50.

string[]

CSS selectors to strip before extraction. Up to 50.

boolean

default:"true"

Strip inline base64 images from the output.

boolean

default:"true"

Remove common ad and tracking elements.

Fetch controls

number

Extra milliseconds to let the page settle before capture. 0–30000.

number

Overall time budget for the request, in milliseconds. 1000–120000.

object

Fetch the page as if from a given region. location.country is an ISO 3166 alpha-2 code (e.g. "us"); location.languages is a list of preferred content languages, most-preferred first (e.g. ["en-US", "en"]).

object

Extra request headers to send when fetching the page.

Caching

number

Serve from cache if a capture of this URL is younger than this many milliseconds. Omit for the default window; 0 always fetches fresh. Capped at 7 days (604800000).

boolean

default:"false"

Only serve from cache. If there is no fresh cached copy, return 404 instead of fetching.

boolean

default:"true"

Persist the capture for later re-extraction.

Async

boolean

default:"false"

Return a job id immediately instead of waiting for the result. Poll GET /v1/web/scrape/{id}.

Response fields

object

Show WebScrapeData

string

The URL you requested.

string

The final URL after any redirects.

number

HTTP status of the fetched page.

boolean

Whether this result was served from cache.

string

Set when the page was returned with a caveat (e.g. partial content).

object

title, pageType (article, listing, forum, docs…), wordCount of the main content, and structure — see below.

string

markdown when the content carries headings, lists and emphasis; plain when it is unstructured prose. plain is not an error: the default extractor optimises for capturing all the content and wins on listing and forum pages. If you need the structure, re-request with preferStructure: true.

object

creditsUsed (charged only on success), creditsRemaining (your balance after this call), and freshness (cache or fresh).

string

Clean Markdown of the main content. Present when markdown is requested.

string

Cleaned main-content HTML. Present when html is requested.

string

The unmodified page HTML. Present when rawHtml is requested.

object

internal and external link lists. Present when links is requested.

object

The page’s own structured data, normalised into typed entities. Present when structured is requested.

string

A concise factual summary. Present when summary is requested.

object

Schema-shaped JSON. Present when json is requested.

Synchronous by default

A scrape waits and returns { "data": ... } with the page. It never degrades to a job id on its own, so one URL means one response shape: the page, or an error explaining why not. The synchronous budget is 90 seconds. A page that runs past it returns 504 with code REQUEST_TIMEOUT. The fetch keeps going in the background, so the capture usually lands in the cache and an immediate retry is fast and cheap.

Set your own client timeout below 90s if you want to give up sooner. Anything above it will never fire, because ours cuts in first.

If you would rather not wait at all, set async: true and get a job id instead:

{ "jobId": "019f3c09-6fae-740f-9257-10c2b6af7f43", "status": "queued" }

Poll it until it finishes:

curl https://api.hydrafetch.com/v1/web/scrape/019f3c09-6fae-740f-9257-10c2b6af7f43 \
  -H "X-API-Key: hf_your_key_here"

const res = await fetch(
  `https://api.hydrafetch.com/v1/web/scrape/${jobId}`,
  { headers: { "X-API-Key": "hf_your_key_here" } },
);
const job = await res.json();

import requests

res = requests.get(
    f"https://api.hydrafetch.com/v1/web/scrape/{job_id}",
    headers={"X-API-Key": "hf_your_key_here"},
)
job = res.json()

The status response carries status (waiting, active, completed, or failed), plus data once completed or error if it failed.

A standard scrape is one credit. LLM-backed formats (summary, json) cost more. Every response reports what it consumed in usage.creditsUsed and your balance in usage.creditsRemaining, and you are only charged on success.

Next steps

Scrape API reference

Full request and response schema with a live playground.

Formats

Markdown, structured data, extracted JSON, and more.

Crawl a whole site

Discover and scrape every page as one job.

Scrape a list of URLs

Provide an explicit set of URLs to scrape together.

Get started

Core concepts

Web Scraping

Web Extraction

When to use

Example request

Example response

Request options

Content

Content filters

Fetch controls

Caching

Async

Response fields

Synchronous by default

Next steps

Scrape API reference

Formats

Crawl a whole site

Scrape a list of URLs

​When to use

​Example request

​Example response

​Request options

​Content

​Content filters

​Fetch controls

​Caching

​Async

​Response fields

​Synchronous by default

​Next steps

Scrape API reference

Formats

Crawl a whole site

Scrape a list of URLs

When to use

Example request

Example response

Request options

Content

Content filters

Fetch controls

Caching

Async

Response fields

Synchronous by default

Next steps