Have an agent build it. Copy the brief below into Claude Code, Cursor, or any coding agent with access to your project. It states the calls, the questions worth asking you first, and the mistakes to avoid.
Agent brief
Agent brief
The pipeline
1. List the URLs without fetching them
map returns a site’s URLs for a single credit, whatever the site’s size. Nothing is fetched, so this is the cheap place to find out how big the job is before committing to it.
search filter keeps only URLs containing a term, which is usually the difference between a corpus and a mess. A documentation site carries a blog, a changelog, and a careers page, and none of them belong in an index you will ask technical questions of.
2. Filter the list yourself
Look at the URLs before you spend anything on content. Drop the pagination, the tag pages, and the archive. This step costs nothing and removes more junk than any cleaning step downstream can. Save the filtered list. It is the input to every future re-run, and it is what you will diff against to notice pages appearing and disappearing.3. Fetch the pages you kept
batch takes up to 25,000 URLs in one call and runs them asynchronously. You get an id back immediately.
GET /v1/web/batch/{id}, or configure a webhook and skip polling entirely.
4. Collapse duplicates before you chunk
The same page is often reachable at several URLs: with and without a trailing slash, with tracking parameters, athttp and https, at /docs/intro and /docs/intro/index.html. Embed all of them and your retriever will happily return the same passage three times and crowd out the answer.
The response gives you what you need to collapse them. Each page carries finalUrl, the URL after any redirects, and redirected telling you whether it moved. Two requested URLs that redirect to one finalUrl are one document.
Deduplicate on finalUrl, not on the URL you submitted. A batch also returns requestedUrl when the URL you sent differed from the normalised one, so you can still map a result back to your input list.
Where two genuinely different URLs return the same content and neither redirects, fall back to a hash of the Markdown. Keep whichever URL is shorter or lives higher in the path, since that is almost always the canonical one a reader would be shown.
5. Chunk with the source attached
Split on headings rather than a fixed character count. The Markdown keeps its heading structure, so a chunk that starts at anh2 is a chunk about one thing, which is what makes retrieval work.
Carry metadata onto every chunk, not just the text: the finalUrl, the page title, the heading path the chunk sits under, and the time you fetched it. A chunk without its source is a passage you cannot cite and cannot invalidate.
Store the raw Markdown alongside the chunks. When you change chunking strategy or embedding model, and you will, re-embedding from stored Markdown costs nothing while re-crawling costs the whole corpus again.
6. Retrieve with citations
Return theurl and title with every retrieved chunk and pass both into the prompt, then tell the model to name which source supports each claim. This is the whole difference between a summary and an answer someone can check.
Show the link in your UI too. A reader who can click through to the page is a reader who can catch the one answer in fifty that is wrong.
Keeping it current
A corpus is only correct on the day you built it. The re-run is the part most pipelines never get around to, and it is three cheap operations. Notice what changed. Re-runmap and diff the URL list against your stored one. New URLs are pages to add. Missing URLs are pages to delete.
Re-fetch without paying for what did not move. Set maxAge in scrapeOptions and a page we already hold, fetched more recently than that, is served without going back to the origin. Store a hash of each page’s Markdown and only re-embed the ones whose hash changed.
Actually delete. This is the step everyone skips. A page removed from the site stays in your index forever unless something removes it, and a retired pricing page or a deprecated API method will keep surfacing in answers long after it stopped being true. Deleting stale chunks matters more than adding new ones, because a missing answer looks like a gap and a wrong answer looks like an answer.
Knowing whether it works
Write down twenty questions your users actually ask, with the page that should answer each one. Run them through retrieval and count how often the right page comes back in the top few results. That number is the only honest measure of whether the corpus is working, and it takes an afternoon to build. Two failure signals are visible in the response itself. A page whosequality reports complete: false came back partial and is worth re-fetching. A page with a wordCount far below its neighbours is usually a login wall, a redirect to a hub page, or a mostly empty template, and it will sit in your index contributing nothing.
Check coverage as well as accuracy. If a whole section of the site never appears in any retrieval result, either nobody asks about it or your filter in step 2 was too aggressive.
What it costs
map is 1 credit regardless of how many URLs come back. batch is 1 credit per page that succeeds. A 400 page corpus is 401 credits, and pages that fail cost nothing.
A weekly refresh over the same 400 pages costs at most another 401, and considerably less when maxAge lets unchanged pages be served without a fresh fetch.