Read PDFs and documents

Have an agent build it. Copy the brief below into Claude Code, Cursor, or any coding agent with access to your project. It states the calls, the questions worth asking you first, and the mistakes to avoid.

Agent brief

Implement this blueprint in my project:
https://docs.hydrafetch.com/blueprints/documents-to-markdown

Read that page, inspect this project's stack, then build the flow end to end.

Add PDF and document reading using the Hydrafetch API.

There is no separate endpoint: POST /v1/web/scrape with {url, formats: ["markdown"]} handles documents the same as web pages, and tables come back as markdown tables. For a set of files, POST /v1/web/batch with the list, which is asynchronous and polled at GET /v1/web/batch/{id}.

Ask me before writing code:
- Where does the list of document URLs come from, and roughly how many and how large? If they are linked from a site, map plus a .pdf filter finds them for 1 credit.
- Do you need the prose, typed fields, or both? Typed fields means a follow-up extract call with a schema.
- What should happen with a scanned document that yields almost no text: OCR it, flag it, or exclude it?
- Where does the output land, and does it need the source URL and heading attached?

The response shape: data with markdown, metadata (title, wordCount) and quality (confidence, complete, blocked).

Compare metadata.wordCount against the document's plausible length and route anything far below it for review, since a scanned PDF returns almost no text without failing. Keep the source url and the heading a passage sat under on every record, because a reader checking a number needs to find it. Store the extracted markdown so nothing has to be re-fetched to re-derive from it. Prefer a webhook over a short polling timeout, since a batch of documents has a long tail.

Notes: authenticate with the X-API-Key header. A document is 1 credit regardless of length, and failures are free. Use batch rather than single calls for more than a handful, since documents take longer to read than pages. Keep the API key on the server and never ship it in client code.

Research lives in PDFs. So do filings, manuals, datasheets, and half the useful pages on a government site. Most scrapers hand you an empty string for all of them. There is no separate document endpoint. Send the URL to scrape and it comes back as Markdown like anything else.

The pipeline

1. Scrape the document like a page

curl -X POST https://api.hydrafetch.com/v1/web/scrape \
  -H "X-API-Key: $HYDRAFETCH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/report.pdf", "formats": ["markdown"]}'

Tables come back as Markdown tables rather than as a column of stray numbers, which is the difference between a document you can query and a wall of text.

2. Do a set of them at once

Documents are slower to read than web pages. Send a list to batch and collect the results when they land, instead of holding a request open per file.

3. Type the contents when you need fields

For a set of filings or datasheets where you want the same fields from each, follow with extract and a schema. Same pipeline as any other page.

Finding the documents in the first place

Usually the PDFs are not the input, they are linked from somewhere. map lists a site’s URLs including the documents, so a filter for .pdf turns a site into a document list for 1 credit. On sites that link documents from pages rather than listing them, scrape the index page with the links format and filter that. search works too when the documents are spread across the web rather than one site, and a domain filter narrows it to the source you trust.

Telling a scan from a document

A PDF that is really a photograph of a page will come back with almost no text, and it will do so without failing. Compare metadata.wordCount against what the document should plausibly hold. A ninety page report that extracted to 200 words is a scanned image, not a short report, and it will sit in your index contributing nothing while looking like a success. Set a floor and route anything under it somewhere a human can see. Whether that means running it through OCR yourself, flagging it, or excluding it is a product decision, but it should be a decision rather than a silent pass. Check the quality object as well. A document reported as incomplete is worth another attempt before you conclude it has nothing in it.

Keeping the provenance

A number pulled out of a filing is worth very little if nobody can find where it came from, and documents are exactly where people want to check. Keep the source URL on every record and every chunk. For a long document, keep the heading the passage sat under too, since “page 40” is not something a reader can search for but a heading is. Store the extracted Markdown, not only what you derived from it. Documents are slower and heavier to fetch than pages, so re-deriving from stored text is meaningfully cheaper than re-fetching a set of large files.

Scale and patience

Documents take longer to read than web pages, and a large one takes noticeably longer. Use batch past a handful of files rather than a loop of single calls, so nothing is waiting on a synchronous response and the work runs in parallel on our side. Expect a wider spread of times than you would with pages. A batch of a hundred documents will have a tail of large ones finishing well after the rest, which is an argument for a webhook over a polling loop with a short timeout. Fetch once and reuse. Where the same document is referenced by several of your records, fetch it once and point them all at the stored text.

What it costs

A document is 1 credit, the same as a page. It costs the same whether the file was two pages or two hundred.

What to watch for

A very large document takes real time to read. Prefer batch over a single call when you have more than a handful, so nothing is waiting on a synchronous response. Check the word count that comes back. A 90 page report that extracted to 200 words is a scanned image of text rather than text, and that is worth knowing before it reaches your index. Keep the source URL and the page it came from. A number pulled out of a filing is worth very little if nobody can find where it came from.

Start here

Get data out of a site

Answer questions and feed agents

Know who you are dealing with

Read PDFs and documents

The pipeline

1. Scrape the document like a page

2. Do a set of them at once

3. Type the contents when you need fields

Finding the documents in the first place

Telling a scan from a document

Keeping the provenance

Scale and patience

What it costs

What to watch for

​The pipeline

​1. Scrape the document like a page

​2. Do a set of them at once

​3. Type the contents when you need fields

​Finding the documents in the first place

​Telling a scan from a document

​Keeping the provenance

​Scale and patience

​What it costs

​What to watch for

The pipeline

1. Scrape the document like a page

2. Do a set of them at once

3. Type the contents when you need fields

Finding the documents in the first place

Telling a scan from a document

Keeping the provenance

Scale and patience

What it costs

What to watch for