Skip to main content
​
Web Scraping

​
Endpoints that only answer a POST

Some data never appears in a page. It sits behind a JSON or GraphQL endpoint that will not answer a GET, and until now the URL was the only thing you could describe.
  • method: "POST" and body send the request the endpoint expects. The body goes verbatim, and you set your own content type through headers, so JSON, form-encoded and anything else are all just a body and a header
  • GET stays the default. Every existing call behaves exactly as before
  • Bodies are capped at 64KB. A request body selects what an endpoint returns; anything larger is a file upload, which is a different thing and not one we do
  • A POST and a GET to the same URL are cached separately, and so are two POSTs with different bodies
Two limits worth knowing. A POST target is fetched, never rendered, because a browser navigation is a GET and rendering one would quietly ask the origin a different question than you did. And PUT, PATCH and DELETE are deliberately not supported: POST is what makes unreachable data reachable, and a narrower surface is easier to reason about for a service that takes arbitrary URLs.
​
MCP

​
Crawl and batch are tools now

Last time these were left off, on the grounds that they are jobs and a tool call is not. That was the wrong call for an agent that is perfectly happy to come back later.
  • crawl and batch start the job and hand back an id; crawl_status and batch_status read it
  • logo joins them, so an agent can fetch a company’s mark without building a URL itself
  • screenshot now takes the same sizing and timing options as the REST call
  • Thirteen tools, all under the existing web:scrape scope, so a grant you have already approved needs no re-authorising
Costs are unchanged: a tool costs exactly what the equivalent REST call costs.
​
Platform

​
Bring your team in

A workspace was effectively one person. It is now a team, with the awkward cases handled rather than left to support.
  • Invite people, set their role, and remove them. A removed member’s seat comes back to the plan immediately
  • Sign up with an email at a company that already has a workspace and you are not turned away: you ask to join, and an owner approves or declines it from the team page
  • An invitation now survives sign-in and sign-up, so following an invite link and then creating an account lands you in the right workspace instead of a new empty one
  • One workspace per company domain, enforced everywhere a workspace can be created
We never show one member another’s email address, including the owner’s.
​
Platform

​
Told in the product, not just by email

Things that only ever arrived in an inbox now appear in the app, as they happen.
  • Credits running low, credits exhausted, a failed payment, a cancelled plan, your logo allowance, a publishable key being used off-domain, and join requests waiting on you
  • They arrive live. No reload, no polling
  • Email is now something you choose per category in settings, split into account alerts and product updates, so you can turn off the newsletter without also turning off “your payment failed”
Account alerts are the ones we would rather you did not miss, which is exactly why they are in the product now as well as your inbox.
​
Brand

​
Colours and marks you can trust

A brand’s palette is only useful if it is the brand’s palette. Several cases where it was not are fixed.
  • The primary colour now comes from the company’s own mark rather than from whatever colour the page happened to use most, and a palette leads with that colour instead of the page background
  • Your operating system’s font is no longer reported as a company’s typeface
  • A mark that cannot say what colour it is loses to one that can, and a saturated mark is classified as mono rather than by brightness alone
  • We never serve an SVG that renders blank, which is what happened when a mark was being coloured by a stylesheet we do not have
  • A domain whose certificate does not cover the bare name is retried on www rather than failing to resolve
  • A styleguide that cannot read one value returns the rest instead of nothing
Logos are cached for a week now rather than a day, so a page rendering the same mark repeatedly stays fast.
​
Brand

​
Logos you can put straight in an <img> tag

A company’s logo as an image URL, with no request to make first and no credits spent.
  • Publishable keys (hf_pk_) are safe to commit and safe to ship in client-side markup. They serve logos and nothing else, so a key lifted from your page source cannot scrape on your account
  • Logo serves never spend credits. They have their own per-cycle allowance, so a page rendering forty logos costs forty serves and zero credits
  • Lock a key to your own domains if you want, and pick the size, theme and shape per URL
  • Served from the edge, so the second visitor gets it immediately
There are drop-in components for React and a browser client if you would rather not build the URL yourself.
​
Web Scraping

​
PDFs come back as Markdown

Point scrape at a PDF and you get Markdown, the same as any page. No separate endpoint, no extra credits, one scrape.
  • Headings stay headings and tables stay tables, instead of collapsing into a flat run of text in reading order
  • Works on anything with a text layer, including exports from Word, LaTeX and design tools
  • Previously a PDF could come back carrying page markup rather than clean Markdown. It no longer does
Scanned PDFs are not supported. A scan is a picture of a document with no text to read, so you get empty markdown, complete: false, confidence: 0 and a warning saying exactly that. Retrying will not change it.
​
Web Extraction

​
A publication date only when the page has one

metadata.publishedTime is now read from what the page actually declares, and is null when it declares nothing.
  • A copyright year in a footer is no longer treated as a publication date. An undated pricing page used to come back “published 1 January” of whatever year the footer mentioned
  • The same applies to the date line at the top of the Markdown: present when the page states one, absent when it does not
  • Pages that declare a real date are unaffected
A wrong date is worse than no date, because you cannot tell it apart from a real one, and anything sorting or filtering on recency quietly inherits the error.
​
Search

​
Search returns results; scraping them is opt-in

search used to fetch every result’s page by default, so a call that reads like one credit billed six. Scraping is now something you ask for.
  • A search costs 1 credit and returns the ranked title, url and snippet
  • Set scrapeResults: true to also fetch each result, at 1 credit per page scraped, the same as scraping that page yourself
  • data on each result is null unless you asked for the scrape
If you were relying on the old default, add scrapeResults: true and the behaviour and cost are exactly what they were.The engine parameter and the data.engine response field are also gone. Which index answered is an implementation detail we route on for cost and availability, and pinning it stopped us falling back when an engine was unavailable.
​
Web Scraping

​
One URL, one response shape

A scrape used to wait, then hand back a job id if the page ran long. That meant a second code path most callers never wrote, appearing only on the rare slow page. It is gone.
  • A single URL is answered inline, always: /scrape, /markdown, /html, /raw-html, /links and /structured
  • The synchronous budget is 90 seconds. Past it you get 504 with code REQUEST_TIMEOUT, naming the budget
  • The fetch keeps running in the background after a timeout, so the capture usually lands in the cache and an immediate retry is fast
  • Want a job instead? Send async: true. It is opt-in now, never automatic
  • Crawl and batch are unchanged: they span many URLs and can genuinely run for minutes
Set your own client timeout below 90 seconds if you want to give up sooner.
​
MCP

​
Every sync endpoint is an MCP tool

The MCP server now exposes the whole synchronous surface, and describes itself so you never have to guess what it can do.
  • New tools: styleguide, screenshot and images, joining scrape, map, search, extract and brand
  • GET /mcp/tools lists every tool with its description, credit cost and full JSON Schema — no authentication required, so a client can see the capabilities before connecting
  • Tools cost exactly what the equivalent REST call costs
  • The new tools sit under the existing web:scrape scope, so an existing grant needs no re-authorising
Crawl and batch stay off MCP for now: they are jobs, and a tool call is not.
​
Brand

​
Brand and styleguide

Answer “what is this company” and “what does it look like” from nothing but a domain, instead of scraping a homepage and guessing.
  • GET /v1/web/brand returns logos suited to light and dark backgrounds, the real brand colours, name, description and social profiles. Every field states whether the company declared it or we harvested it
  • POST /v1/web/brand resolves up to 50 domains in one call
  • GET /v1/web/brand/search finds brands we already hold by company name or domain, so you can turn what a user typed into a domain before resolving it. Returns enough to render a picker, never resolves a new domain, and is not billed
  • GET /v1/web/styleguide reads the design system a site actually paints, from computed styles in a real browser, so a colour defined through four layers of CSS variables comes back as the hex it resolved to. Colours arrive by role with contrast ratios, so they drop into a theme without guesswork
A brand resolves once and is served from storage after that, so the second call for a domain is the same price and immediate.
​
Platform

​
Billing that matches what you got

Several cases where a call could be charged without delivering a usable page are now not billed at all.
  • An origin’s own error page is no longer recorded as a delivered scrape
  • A response with no bytes is never treated as a success, whatever status the origin returned
  • A failed render no longer discards the static capture it was meant to improve on
  • Upstream failures surface as errors rather than empty 200s
Failures were always meant to be free. These were the paths where that promise leaked.
​
Web Scraping

​
Ask for structure when you need it

preferStructure keeps headings, lists and tables as real markdown. It was already on the REST API; it is now on the MCP tools too, so agents can ask for it.
  • metadata.format is now metadata.structure, reporting markdown or plain
  • plain is not a failure — the default extractor optimises for capturing every word and wins on listing and forum pages — but if you wanted headings, retry with preferStructure: true
  • Available on scrape, search and extract over both REST and MCP
​
Web Scraping

​
See whether your webhooks landed

Every callback we attempt for a crawl or batch is now recorded and readable, so a webhook that never arrives is something you can diagnose instead of guess at.
  • GET /v1/web/crawl/{id}/deliveries and GET /v1/web/batch/{id}/deliveries
  • Each delivery reports its status, attempt count, the HTTP status your endpoint returned, and why the last attempt failed
  • A failed row with a 5xx means we couldn’t reach you; a 4xx means we did and you rejected it
​
Platform

​
Errors you can branch on

Every error now returns the same envelope with a stable, machine-readable code. Branch on the code — the wording of message may change, the code won’t. Each response also carries a requestId to quote if you need us to look something up.
​
Web Extraction

​
Trustworthy extraction

Structured extraction now returns, per field, how confident it is and the exact passage a value came from. Turn it on with showConfidence and showSources.
  • Extract across a whole site: pass a /* scope in urls and every discovered page is merged into one result.
  • New entities primitive returns normalized people, companies, and products found on a page.
  • Every response reports the credits it consumed and how fresh the data is.
​
Web Scraping

​
Dedicated format endpoints

Prefer a single, purpose-built route? Each output now has its own endpoint, alongside the flexible /scrape.
  • POST /v1/web/markdown for clean Markdown
  • POST /v1/web/html for cleaned main-content HTML
  • POST /v1/web/raw-html for unmodified page HTML
  • POST /v1/web/links for a page’s links
  • POST /v1/web/structured for the page’s own structured data
They share every option and response shape with /scrape, so you can switch anytime.
​
Web Scraping

​
Search, crawl, and media

  • Search the web and get ranked results, each scraped to clean data. Filter by time, domain, and country.
  • Crawl a whole site as one asynchronous job, with completion webhooks so you do not have to poll.
  • Media: capture a full-page screenshot or collect a page’s images.
​
Platform

​
Launch

The first public version of the Hydrafetch API.
  • POST /v1/web/scrape takes one URL and returns clean Markdown and structured data.
  • POST /v1/web/map discovers every URL on a site.
  • Credit-based billing, charged only on success, with a live balance on every response.
  • API-key authentication through the X-API-Key header.