Jobs & webhooks

Some calls finish in the moment; others run long enough to hand back a job you follow up on. Hydrafetch keeps the shape predictable: a synchronous result comes back as { data }, and a job comes back as an id you poll — or a webhook that calls you when it’s done.

Scrape: synchronous by default

A scrape waits for the result and returns it inline. You get { data } with the formats you asked for.

{
  "data": {
    "url": "https://example.com",
    "status": 200,
    "markdown": "# Example Domain\n\n..."
  }
}

If a scrape runs past the synchronous wait — or if you set async: true — it returns a job id instead, with a 201:

{
  "jobId": "019f3c09-6fae-740f-9257-10c2b6af7f43",
  "status": "queued"
}

Poll it until it’s done:

curl https://api.hydrafetch.com/v1/web/scrape/019f3c09-6fae-740f-9257-10c2b6af7f43 \
  -H "X-API-Key: hf_your_key_here"

const res = await fetch(
  "https://api.hydrafetch.com/v1/web/scrape/019f3c09-6fae-740f-9257-10c2b6af7f43",
  { headers: { "X-API-Key": "hf_your_key_here" } },
);
const job = await res.json();

The status response reports the job state and, once finished, the page:

string

waiting, active, completed, or failed.

object

The scraped page, present when status is completed.

string

Set when status is failed.

Set async: true up front when you know a page is heavy — a JavaScript-rendered app, a long settle time — so your request never blocks waiting for it.

Crawl and batch: always async

A crawl (discover and scrape a whole site) and a batch (scrape an explicit list of URLs) always run as background jobs. Starting one returns an id immediately with a 201:

{
  "crawlId": "019f3c09-6fae-740f-9257-10c2b6af7f43",
  "status": "queued"
}

{
  "batchId": "019f3c09-6fae-740f-9257-10c2b6af7f43",
  "status": "queued"
}

You then track it one of two ways.

Poll
Webhook

Request the job’s status endpoint for live progress and per-page results:

curl https://api.hydrafetch.com/v1/web/crawl/019f3c09-6fae-740f-9257-10c2b6af7f43 \
  -H "X-API-Key: hf_your_key_here"

Batches use GET /v1/web/batch/{id}. Both return the same status shape:

{
  "id": "019f3c09-6fae-740f-9257-10c2b6af7f43",
  "kind": "crawl",
  "status": "running",
  "total": 100,
  "completed": 42,
  "failed": 1,
  "creditsUsed": 42,
  "pages": [
    { "url": "https://example.com/blog/post", "status": "completed", "depth": 1, "data": { } }
  ]
}

status moves through running to completed (or failed / cancelled). Each entry in pages carries its own status and, once done, its scraped data.

Register a webhook when you start the job and Hydrafetch calls you as it progresses — including the full status when it finishes — so you don’t have to poll.

curl -X POST https://api.hydrafetch.com/v1/web/crawl \
  -H "X-API-Key: hf_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "limit": 100,
    "webhook": {
      "url": "https://your-app.com/hooks/hydrafetch",
      "events": ["page", "completed"],
      "secret": "whsec_a_long_random_string",
      "headers": { "Authorization": "Bearer your-token" }
    }
  }'

Two events fire:

Event	When
`crawl.page` / `batch.page`	Each time a page finishes — carries that page’s `status` and scraped `data`.
`crawl.completed` / `batch.completed`	Once, when the job reaches a terminal state — carries the full status.

string

required

Where to POST events. Must be a public http(s) URL — private, loopback, and link-local addresses are rejected with a 400.

string[]

Which events to receive: page, completed, or both. Defaults to both. Use ["completed"] if you only want the final result and not a call per page.

object

Extra headers sent with each callback — use them to authenticate the request to your endpoint.

string

A shared secret. When set, every callback is signed so you can verify it really came from Hydrafetch. See Verifying a webhook.

Deliveries are durable. If your endpoint is down or returns a non-2xx, we retry with exponential backoff (up to 6 attempts) rather than dropping the event.

The webhook and polling aren’t exclusive. You can register a webhook and still poll the status endpoint whenever you want.

Verifying a webhook

Your webhook endpoint is a public URL, so anyone who finds it can POST to it. Set a webhook.secret and we sign every callback, letting you prove a request came from us and reject anything that didn’t. Each signed delivery carries these headers:

Header	What it is
`X-Hydrafetch-Signature`	`t=<timestamp>,v1=<hmac>` — HMAC-SHA256 of `<timestamp>.<raw body>`, keyed with your secret
`X-Hydrafetch-Timestamp`	Unix seconds at send time
`X-Hydrafetch-Event`	The event, e.g. `crawl.completed`
`X-Hydrafetch-Delivery`	Unique id for this delivery attempt

To verify, recompute the HMAC over the raw request body — not a re-serialized version of the parsed JSON, which may produce different bytes and never match:

import hashlib, hmac, time

def verify(raw_body: bytes, signature_header: str, secret: str, tolerance_s: int = 300) -> bool:
    parts = dict(p.split("=", 1) for p in signature_header.split(","))
    timestamp, received = parts["t"], parts["v1"]

    # Reject anything too old — this is what stops a captured call being replayed at you later.
    if abs(time.time() - int(timestamp)) > tolerance_s:
        return False

    expected = hmac.new(
        secret.encode(), f"{timestamp}.".encode() + raw_body, hashlib.sha256
    ).hexdigest()
    return hmac.compare_digest(expected, received)  # constant-time

Compare signatures in constant time (hmac.compare_digest, crypto.timingSafeEqual) — a plain == leaks timing information. And always check the timestamp: a signature alone stays valid forever, so without a freshness window a captured delivery can be replayed at you.

The timestamp is signed along with the body, so an attacker cannot take a real delivery and re-send it under a fresh timestamp. If you don’t set a secret, callbacks are still sent — just unsigned.

Checking whether a delivery landed

A webhook that never arrives is the hardest kind of bug to chase from your side, so we keep the delivery history and expose it. Every callback we attempted for a job is listed, with its status, how many attempts we made, and what your endpoint returned last.

curl https://api.hydrafetch.com/v1/web/crawl/019f3c09-6fae-740f-9257-10c2b6af7f43/deliveries \
  -H "X-API-Key: $HYDRAFETCH_API_KEY"

Batches use the same shape at GET /v1/web/batch/{id}/deliveries.

object[]

Show WebhookDelivery

string

Delivery id.

string

The event this delivery carried.

string

pending while retries remain, success once delivered, failed once we gave up.

number

How many delivery attempts have been made.

number | null

The HTTP status your endpoint returned on the last attempt.

string | null

Why the last attempt failed, if it did.

string | null

When the delivery succeeded (ISO 8601).

string

When the delivery was first created (ISO 8601).

A failed row with a responseStatus of 5xx usually means your endpoint was down when we called; one with a 4xx means we reached you and you rejected it — check the signature verification above before assuming the payload is wrong.

Which calls return what

Call	Response	If you ask for a job instead
Scrape	`{ data }` inline, or `504` past 90s	`async: true` returns `{ jobId, status }` — poll `GET /v1/web/scrape/{id}`
Map, Search, Extract	`{ data }` inline	—
Crawl	Always `{ crawlId, status }`	Poll `GET /v1/web/crawl/{id}` or register a webhook
Batch	Always `{ batchId, status }`	Poll `GET /v1/web/batch/{id}` or register a webhook

A single URL is always answered inline unless you opt out. Crawl and batch are jobs because they span many URLs and can run for minutes; one page cannot, so it does not get a second response shape you have to write code for.

Next: Errors

The structured error model and every status you’ll see.

Get started

Core concepts

Web Scraping

Web Extraction

Scrape: synchronous by default

Crawl and batch: always async

Verifying a webhook

Checking whether a delivery landed

Which calls return what

Next: Errors

​Scrape: synchronous by default

​Crawl and batch: always async

​Verifying a webhook

​Checking whether a delivery landed

​Which calls return what

Next: Errors

Scrape: synchronous by default

Crawl and batch: always async

Verifying a webhook

Checking whether a delivery landed

Which calls return what