Crawl

Crawl points Hydrafetch at a starting URL, discovers the site’s pages for you, and scrapes each one. It runs as a single asynchronous job: you get a crawlId back immediately, then poll it or register a webhook to collect the per-page results. One credit per page scraped.

When to use

You want every page of a site (or a section of it) as clean data, and you do not have the URL list yourself.
The job is large enough that waiting on a single request is impractical.

If you already have the exact URLs, use Batch — no discovery needed. To preview which URLs a crawl would reach without scraping them, use Map.

Example request

Send a POST to /v1/web/crawl. Scope the crawl with limit, maxDepth, and path filters, and control how each page is scraped with scrapeOptions.

curl -X POST https://api.hydrafetch.com/v1/web/crawl \
  -H "X-API-Key: hf_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "limit": 100,
    "maxDepth": 2,
    "includePaths": ["^/blog/.*"],
    "excludePaths": ["^/tag/.*"],
    "scrapeOptions": { "formats": ["markdown", "links"] }
  }'

const res = await fetch("https://api.hydrafetch.com/v1/web/crawl", {
  method: "POST",
  headers: {
    "X-API-Key": "hf_your_key_here",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    url: "https://example.com",
    limit: 100,
    maxDepth: 2,
    includePaths: ["^/blog/.*"],
    excludePaths: ["^/tag/.*"],
    scrapeOptions: { formats: ["markdown", "links"] },
  }),
});
const { crawlId } = await res.json();

import requests

res = requests.post(
    "https://api.hydrafetch.com/v1/web/crawl",
    headers={"X-API-Key": "hf_your_key_here"},
    json={
        "url": "https://example.com",
        "limit": 100,
        "maxDepth": 2,
        "includePaths": ["^/blog/.*"],
        "excludePaths": ["^/tag/.*"],
        "scrapeOptions": {"formats": ["markdown", "links"]},
    },
)
crawl_id = res.json()["crawlId"]

Example response

The crawl is accepted right away:

{ "crawlId": "019f3c09-6fae-740f-9257-10c2b6af7f43", "status": "queued" }

Request options

string

required

The site to start from. Must be http(s).

number

default:"100"

Maximum number of pages to scrape. 1–5000.

number

How many links deep from the starting page to follow. 0–10.

string[]

Only follow URLs whose path matches every one of these patterns (e.g. ["^/blog/.*"]). Up to 50.

string[]

Skip URLs whose path matches any of these patterns (e.g. ["^/tag/.*"]). Up to 50.

boolean

default:"false"

Also follow links into subdomains of the starting site.

boolean

default:"false"

Also follow links that lead off the starting site.

boolean

default:"false"

Treat URLs that differ only by query string as the same page.

string

default:"include"

Whether to seed discovery from the site’s published page list: skip or include.

object

How to scrape each page — formats, onlyMainContent, includeTags, excludeTags, removeBase64Images, blockAds, renderJs, waitFor, timeout, location, headers, preferStructure, maxAge. Same options as a single Scrape, with per-page formats limited to markdown, html, rawHtml, links, and structured.

object

Register a callback instead of polling. webhook.url receives progress and completion events; webhook.headers are extra headers sent with each callback (e.g. for authentication).

Poll for results

Poll GET /v1/web/crawl/{id} for progress and per-page results.

curl https://api.hydrafetch.com/v1/web/crawl/019f3c09-6fae-740f-9257-10c2b6af7f43 \
  -H "X-API-Key: hf_your_key_here"

const res = await fetch(
  `https://api.hydrafetch.com/v1/web/crawl/${crawlId}`,
  { headers: { "X-API-Key": "hf_your_key_here" } },
);
const status = await res.json();

import requests

res = requests.get(
    f"https://api.hydrafetch.com/v1/web/crawl/{crawl_id}",
    headers={"X-API-Key": "hf_your_key_here"},
)
status = res.json()

{
  "id": "019f3c09-6fae-740f-9257-10c2b6af7f43",
  "kind": "crawl",
  "status": "running",
  "seedUrl": "https://example.com",
  "total": 100,
  "completed": 42,
  "failed": 1,
  "creditsUsed": 42,
  "pages": [
    {
      "url": "https://example.com/blog/post",
      "status": "completed",
      "depth": 1,
      "error": null,
      "data": {
        "url": "https://example.com/blog/post",
        "finalUrl": "https://example.com/blog/post",
        "status": 200,
        "cached": false,
        "metadata": { "title": "A post", "pageType": "article", "wordCount": 812 },
        "usage": { "creditsUsed": 1, "creditsRemaining": 4993, "freshness": "fresh" },
        "markdown": "# A post\n\n..."
      }
    }
  ]
}

Response fields

string

The crawl id.

string

crawl or batch.

string

Overall job state: running, completed, failed, or cancelled.

string

The starting URL.

number

Total pages in this job.

number

Pages scraped so far.

number

Pages that failed.

number

Credits consumed so far. One per scraped page.

object[]

Per-page results, each with url, status (queued, running, completed, failed), depth, error, and data (the scraped page, once completed).

A crawl costs one credit per page scraped, reflected in creditsUsed. Scope the job with limit, maxDepth, and path filters to keep spend predictable, and preview reach first with Map.

Next steps

Crawl API reference

Full request and response schema with a live playground.

Map a site

Preview a crawl’s scope without scraping.

Scrape a list of URLs

When you already have the URLs.

Scrape one URL

The per-page primitive behind a crawl.

Get started

Core concepts

Web Scraping

Web Extraction

When to use

Example request

Example response

Request options

Poll for results

Response fields

Next steps

Crawl API reference

Map a site

Scrape a list of URLs

Scrape one URL

​When to use

​Example request

​Example response

​Request options

​Poll for results

​Response fields

​Next steps

Crawl API reference

Map a site

Scrape a list of URLs

Scrape one URL

When to use

Example request

Example response

Request options

Poll for results

Response fields

Next steps