Crawl

Crawl points Hydrafetch at a starting URL, discovers the site’s pages for you, and scrapes each one. It runs as a single asynchronous job: you get a crawlId back immediately, then poll it or register a webhook to collect the per-page results. One credit per page scraped.

When to use

You want every page of a site (or a section of it) as clean data, and you do not have the URL list yourself.
The job is large enough that waiting on a single request is impractical.

If you already have the exact URLs, use Batch — no discovery needed. To preview which URLs a crawl would reach without scraping them, use Map.

Example request

Send a POST to /v1/web/crawl. Scope the crawl with limit, maxDepth, and path filters, and control how each page is scraped with scrapeOptions.

curl -X POST https://api.hydrafetch.com/v1/web/crawl \
  -H "X-API-Key: hf_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/blog",
    "limit": 100,
    "maxDepth": 2,
    "includePaths": ["^/blog"],
    "excludePaths": ["^/tag"],
    "scrapeOptions": { "formats": ["markdown", "links"] }
  }'

const res = await fetch("https://api.hydrafetch.com/v1/web/crawl", {
  method: "POST",
  headers: {
    "X-API-Key": "hf_your_key_here",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    url: "https://example.com/blog",
    limit: 100,
    maxDepth: 2,
    includePaths: ["^/blog"],
    excludePaths: ["^/tag"],
    scrapeOptions: { formats: ["markdown", "links"] },
  }),
});
const { crawlId } = await res.json();

import requests

res = requests.post(
    "https://api.hydrafetch.com/v1/web/crawl",
    headers={"X-API-Key": "hf_your_key_here"},
    json={
        "url": "https://example.com/blog",
        "limit": 100,
        "maxDepth": 2,
        "includePaths": ["^/blog"],
        "excludePaths": ["^/tag"],
        "scrapeOptions": {"formats": ["markdown", "links"]},
    },
)
crawl_id = res.json()["crawlId"]

Example response

The crawl is accepted right away:

{ "crawlId": "019f3c09-6fae-740f-9257-10c2b6af7f43", "status": "queued" }

Keeping a crawl inside one section

A starting URL says where to begin, not where to stay. Discovery also uses the site’s own published page list, which covers everything, so pointing a crawl at https://example.com/blog and setting nothing else will happily return pages from the rest of the site. Add includePaths to hold it inside the section, and write the pattern so that the starting page matches it too:

{ "url": "https://example.com/blog", "includePaths": ["^/blog"] }

["^/blog/.*"] looks equivalent and is not. A starting URL loses its trailing slash, so https://example.com/blog/ starts at the path /blog, which that pattern does not match. The crawl would then have no page to begin from, so we refuse the request instead of handing back a job that finishes with nothing in it.

Request options

string

required

The site to start from. Must be http(s).

number

default:"100"

Maximum number of pages to scrape. 1–500.

number

How many links deep from the starting page to follow. 0–10.

string[]

Only follow URLs whose path matches every one of these patterns (e.g. ["^/blog"]). Up to 50. The patterns apply to the starting URL as well. If they exclude it, the crawl has nowhere to begin, so the request is refused rather than returning an empty job.

string[]

Skip URLs whose path matches any of these patterns (e.g. ["^/tag"]). Up to 50. As with includePaths, these apply to the starting URL too.

boolean

default:"false"

Also follow links into subdomains of the starting site.

boolean

default:"false"

Also follow links that lead off the starting site.

boolean

default:"false"

Treat URLs that differ only by query string as the same page.

string

default:"include"

Whether to seed discovery from the site’s published page list: skip or include. Because that list covers the whole site, starting from a section does not by itself limit the crawl to that section. Use includePaths to do that.

object

How to scrape each page — formats, onlyMainContent, includeTags, excludeTags, removeBase64Images, blockAds, waitFor, timeout, location, headers, preferStructure, maxAge. Same options as a single Scrape, with per-page formats limited to markdown, html, rawHtml, links, and structured.

object

Register a callback instead of polling. webhook.url receives progress and completion events; webhook.headers are extra headers sent with each callback (e.g. for authentication).

Poll for results

Poll GET /v1/web/crawl/{id} for progress and per-page results.

curl https://api.hydrafetch.com/v1/web/crawl/019f3c09-6fae-740f-9257-10c2b6af7f43 \
  -H "X-API-Key: hf_your_key_here"

const res = await fetch(
  `https://api.hydrafetch.com/v1/web/crawl/${crawlId}`,
  { headers: { "X-API-Key": "hf_your_key_here" } },
);
const status = await res.json();

import requests

res = requests.get(
    f"https://api.hydrafetch.com/v1/web/crawl/{crawl_id}",
    headers={"X-API-Key": "hf_your_key_here"},
)
status = res.json()

{
  "id": "019f3c09-6fae-740f-9257-10c2b6af7f43",
  "kind": "crawl",
  "status": "running",
  "seedUrl": "https://example.com",
  "total": 100,
  "completed": 42,
  "failed": 1,
  "creditsUsed": 42,
  "pages": [
    {
      "url": "https://example.com/blog/post",
      "status": "completed",
      "depth": 1,
      "discoveredVia": "sitemap",
      "error": null,
      "data": {
        "url": "https://example.com/blog/post",
        "finalUrl": "https://example.com/blog/post",
        "status": 200,
        "cached": false,
        "metadata": { "title": "A post", "pageType": "article", "wordCount": 812 },
        "usage": { "creditsUsed": 1, "creditsRemaining": 4993, "freshness": "fresh" },
        "markdown": "# A post\n\n..."
      }
    }
  ]
}

Response fields

string

The crawl id.

string

crawl or batch.

string

Overall job state: running, completed, failed, or cancelled.

string

The starting URL.

number

Total pages in this job.

number

Pages scraped so far.

number

Pages that failed.

number

Credits consumed so far. One per scraped page.

object[]

Per-page results, each with url, status (queued, running, completed, failed), depth, discoveredVia, error, and data (the scraped page, once completed).discoveredVia tells you how a URL entered the job: requested for one you gave us (the crawl seed, or any batch entry), sitemap for one read from the site’s sitemap, link for one followed from a page we had already fetched. It is the quickest way to see why a crawl returned the set it did, and whether raising the limit or the depth would find more.

A crawl costs one credit per page scraped, reflected in creditsUsed. Scope the job with limit, maxDepth, and path filters to keep spend predictable, and preview reach first with Map.

Next steps

Crawl API reference

Full request and response schema with a live playground.

Map a site

Preview a crawl’s scope without scraping.

Scrape a list of URLs

When you already have the URLs.

Scrape one URL

The per-page primitive behind a crawl.

Get started

Core concepts

Web Scraping

Web Extraction

When to use

Example request

Example response

Keeping a crawl inside one section

Request options

Poll for results

Response fields

Next steps

Crawl API reference

Map a site

Scrape a list of URLs

Scrape one URL

​When to use

​Example request

​Example response

​Keeping a crawl inside one section

​Request options

​Poll for results

​Response fields

​Next steps

Crawl API reference

Map a site

Scrape a list of URLs

Scrape one URL

When to use

Example request

Example response

Keeping a crawl inside one section

Request options

Poll for results

Response fields

Next steps