> ## Documentation Index
> Fetch the complete documentation index at: https://docs.hydrafetch.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Jobs & webhooks
> When calls return immediately, when they run as background jobs, and how to get results by polling or webhook.
Some calls finish in the moment; others run long enough to hand back a job you follow up on. Hydrafetch keeps the shape predictable: a synchronous result comes back as `{ data }`, and a job comes back as an id you poll — or a webhook that calls you when it's done.
## Scrape: synchronous by default
A scrape waits for the result and returns it inline. You get `{ data }` with the formats you asked for.
```json theme={"dark"}
{
"data": {
"url": "https://example.com",
"status": 200,
"markdown": "# Example Domain\n\n..."
}
}
```
If a scrape runs past the synchronous wait — or if you set `async: true` — it returns a job id instead, with a `201`:
```json theme={"dark"}
{
"jobId": "019f3c09-6fae-740f-9257-10c2b6af7f43",
"status": "queued"
}
```
Poll it until it's done:
```bash cURL theme={"dark"}
curl https://api.hydrafetch.com/v1/web/scrape/019f3c09-6fae-740f-9257-10c2b6af7f43 \
-H "X-API-Key: hf_your_key_here"
```
```javascript Node theme={"dark"}
const res = await fetch(
"https://api.hydrafetch.com/v1/web/scrape/019f3c09-6fae-740f-9257-10c2b6af7f43",
{ headers: { "X-API-Key": "hf_your_key_here" } },
);
const job = await res.json();
```
The status response reports the job state and, once finished, the page:
`waiting`, `active`, `completed`, or `failed`.
The scraped page, present when `status` is `completed`.
Set when `status` is `failed`.
Set `async: true` up front when you know a page is heavy — a JavaScript-rendered app, a long settle time — so your request never blocks waiting for it.
## Crawl and batch: always async
A crawl (discover and scrape a whole site) and a batch (scrape an explicit list of URLs) always run as background jobs. Starting one returns an id immediately with a `201`:
```json Crawl theme={"dark"}
{
"crawlId": "019f3c09-6fae-740f-9257-10c2b6af7f43",
"status": "queued"
}
```
```json Batch theme={"dark"}
{
"batchId": "019f3c09-6fae-740f-9257-10c2b6af7f43",
"status": "queued"
}
```
You then track it one of two ways.
Request the job's status endpoint for live progress and per-page results:
```bash theme={"dark"}
curl https://api.hydrafetch.com/v1/web/crawl/019f3c09-6fae-740f-9257-10c2b6af7f43 \
-H "X-API-Key: hf_your_key_here"
```
Batches use `GET /v1/web/batch/{id}`. Both return the same status shape:
```json theme={"dark"}
{
"id": "019f3c09-6fae-740f-9257-10c2b6af7f43",
"kind": "crawl",
"status": "running",
"total": 100,
"completed": 42,
"failed": 1,
"creditsUsed": 42,
"pages": [
{ "url": "https://example.com/blog/post", "status": "completed", "depth": 1, "data": { } }
]
}
```
`status` moves through `running` to `completed` (or `failed` / `cancelled`). Each entry in `pages` carries its own `status` and, once done, its scraped `data`.
Register a `webhook` when you start the job and Hydrafetch calls you as it progresses — including the full status when it finishes — so you don't have to poll.
```bash theme={"dark"}
curl -X POST https://api.hydrafetch.com/v1/web/crawl \
-H "X-API-Key: hf_your_key_here" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com",
"limit": 100,
"webhook": {
"url": "https://your-app.com/hooks/hydrafetch",
"events": ["page", "completed"],
"secret": "whsec_a_long_random_string",
"headers": { "Authorization": "Bearer your-token" }
}
}'
```
Two events fire:
| Event | When |
| ------------------------------------- | ---------------------------------------------------------------------------- |
| `crawl.page` / `batch.page` | Each time a page finishes — carries that page's `status` and scraped `data`. |
| `crawl.completed` / `batch.completed` | Once, when the job reaches a terminal state — carries the full status. |
Where to POST events. Must be a **public** http(s) URL — private, loopback, and link-local addresses are rejected with a `400`.
Which events to receive: `page`, `completed`, or both. Defaults to both. Use `["completed"]` if you only want the final result and not a call per page.
Extra headers sent with each callback — use them to authenticate the request to your endpoint.
A shared secret. When set, every callback is signed so you can verify it really came from Hydrafetch. See [Verifying a webhook](#verifying-a-webhook).
Deliveries are durable. If your endpoint is down or returns a non-2xx, we retry with exponential backoff (up to 6 attempts) rather than dropping the event.
The webhook and polling aren't exclusive. You can register a webhook and still poll the status endpoint whenever you want.
## Verifying a webhook
Your webhook endpoint is a public URL, so anyone who finds it can POST to it. Set a `webhook.secret` and we sign every callback, letting you prove a request came from us and reject anything that didn't.
Each signed delivery carries these headers:
| Header | What it is |
| ------------------------ | ------------------------------------------------------------------------------------------- |
| `X-Hydrafetch-Signature` | `t=,v1=` — HMAC-SHA256 of `.`, keyed with your secret |
| `X-Hydrafetch-Timestamp` | Unix seconds at send time |
| `X-Hydrafetch-Event` | The event, e.g. `crawl.completed` |
| `X-Hydrafetch-Delivery` | Unique id for this delivery attempt |
To verify, recompute the HMAC over the **raw request body** — not a re-serialized version of the parsed JSON, which may produce different bytes and never match:
```python theme={"dark"}
import hashlib, hmac, time
def verify(raw_body: bytes, signature_header: str, secret: str, tolerance_s: int = 300) -> bool:
parts = dict(p.split("=", 1) for p in signature_header.split(","))
timestamp, received = parts["t"], parts["v1"]
# Reject anything too old — this is what stops a captured call being replayed at you later.
if abs(time.time() - int(timestamp)) > tolerance_s:
return False
expected = hmac.new(
secret.encode(), f"{timestamp}.".encode() + raw_body, hashlib.sha256
).hexdigest()
return hmac.compare_digest(expected, received) # constant-time
```
Compare signatures in constant time (`hmac.compare_digest`, `crypto.timingSafeEqual`) — a plain `==` leaks timing information. And always check the timestamp: a signature alone stays valid forever, so without a freshness window a captured delivery can be replayed at you.
The timestamp is signed along with the body, so an attacker cannot take a real delivery and re-send it under a fresh timestamp. If you don't set a secret, callbacks are still sent — just unsigned.
## Which calls return what
| Call | Default | Long-running |
| -------------------- | ---------------------------- | ---------------------------------------------------- |
| Scrape | `{ data }` inline | `{ jobId, status }` — poll `GET /v1/web/scrape/{id}` |
| Crawl | Always `{ crawlId, status }` | Poll `GET /v1/web/crawl/{id}` or register a webhook |
| Batch | Always `{ batchId, status }` | Poll `GET /v1/web/batch/{id}` or register a webhook |
| Map, Search, Extract | `{ data }` inline | — |
The structured error model and every status you'll see.