Have an agent build it. Copy the brief below into Claude Code, Cursor, or any coding agent with access to your project. It states the calls, the questions worth asking you first, and the mistakes to avoid.
Agent brief
Agent brief
extract takes URLs and a schema and returns records shaped the way you asked, merged across pages.
The pipeline
1. Find the pages that hold the rows
Usemap with a search filter to pull just the listing URLs, for one credit and without fetching anything.
2. Describe the fields, in a schema
3. Let a path pattern do the fan-out
A URL ending in/* marks a crawl scope. Every page discovered under that path is extracted and merged into one result set, so a small list can cover a large section.
Give every row a stable identity
A one-off extraction produces a list. A pipeline that runs again produces the same list with small differences, and without an identity you cannot tell an updated row from a new one. Derive the id from the page, not from the content. The URL is stable while a product name, a price, and a description all change. Normalise it first (drop tracking parameters, settle on one trailing slash convention) and hash it, or store the URL itself as the key. Then an update is an upsert rather than a duplicate, and you can keep a history: same id, new price, new timestamp. That history is usually worth more than the current snapshot, because it is the only way to answer when something changed.When the shape silently breaks
This is the failure mode that costs people weeks, because nothing errors. A site redesigns. The price now lives in a different element. Extraction still succeeds, still returns valid JSON, andpriceUsd is now null on every page. The pipeline is green, the rows are wrong, and nobody notices until someone asks why the report looks odd.
Track the fill rate per field on every run, meaning the share of records where the field came back non-null. Store it. When a field that was 97% full for three months drops to 4% overnight, that is a site change, not a data change, and it should page someone.
Set a floor and fail loudly under it. A run where a required field is missing from most records should stop rather than write, because a partial overwrite of good data is much harder to undo than a run that refused to start.
Spot-check a handful of records against the live page after any run that looks unusual. Keeping the source URL on every record is what makes that a thirty second check instead of an investigation.
Keeping it current
Re-discovery and re-extraction are separate decisions, and they have very different prices. Re-runmap often, since it is 1 credit and tells you which URLs are new, which disappeared, and therefore which products were added or retired. Most of the value of a repeat run is in that diff alone.
Re-extract selectively. Extraction is the expensive step, so do not re-run it across the whole catalogue on a schedule. Extract new URLs always, and re-extract existing ones on a slower cycle or when a cheap signal suggests the page moved, such as a scrape of the page returning different Markdown from last time.
Delete what left. A product removed from the site should be marked gone rather than silently retained, or your dataset slowly fills with items nobody can buy.