When calls return immediately, when they run as background jobs, and how to get results by polling or webhook.
Some calls finish in the moment; others run long enough to hand back a job you follow up on. Hydrafetch keeps the shape predictable: a synchronous result comes back as { data }, and a job comes back as an id you poll — or a webhook that calls you when it’s done.
A crawl (discover and scrape a whole site) and a batch (scrape an explicit list of URLs) always run as background jobs. Starting one returns an id immediately with a 201:
status moves through running to completed (or failed / cancelled). Each entry in pages carries its own status and, once done, its scraped data.
Register a webhook when you start the job and Hydrafetch calls you as it progresses — including the full status when it finishes — so you don’t have to poll.
A shared secret. When set, every callback is signed so you can verify it really came from Hydrafetch. See Verifying a webhook.
Deliveries are durable. If your endpoint is down or returns a non-2xx, we retry with exponential backoff (up to 6 attempts) rather than dropping the event.
The webhook and polling aren’t exclusive. You can register a webhook and still poll the status endpoint whenever you want.
Your webhook endpoint is a public URL, so anyone who finds it can POST to it. Set a webhook.secret and we sign every callback, letting you prove a request came from us and reject anything that didn’t.Each signed delivery carries these headers:
Header
What it is
X-Hydrafetch-Signature
t=<timestamp>,v1=<hmac> — HMAC-SHA256 of <timestamp>.<raw body>, keyed with your secret
X-Hydrafetch-Timestamp
Unix seconds at send time
X-Hydrafetch-Event
The event, e.g. crawl.completed
X-Hydrafetch-Delivery
Unique id for this delivery attempt
To verify, recompute the HMAC over the raw request body — not a re-serialized version of the parsed JSON, which may produce different bytes and never match:
import hashlib, hmac, timedef verify(raw_body: bytes, signature_header: str, secret: str, tolerance_s: int = 300) -> bool: parts = dict(p.split("=", 1) for p in signature_header.split(",")) timestamp, received = parts["t"], parts["v1"] # Reject anything too old — this is what stops a captured call being replayed at you later. if abs(time.time() - int(timestamp)) > tolerance_s: return False expected = hmac.new( secret.encode(), f"{timestamp}.".encode() + raw_body, hashlib.sha256 ).hexdigest() return hmac.compare_digest(expected, received) # constant-time
Compare signatures in constant time (hmac.compare_digest, crypto.timingSafeEqual) — a plain == leaks timing information. And always check the timestamp: a signature alone stays valid forever, so without a freshness window a captured delivery can be replayed at you.
The timestamp is signed along with the body, so an attacker cannot take a real delivery and re-send it under a fresh timestamp. If you don’t set a secret, callbacks are still sent — just unsigned.
