Skip to main content
Have an agent build it. Copy the brief below into Claude Code, Cursor, or any coding agent with access to your project. It states the calls, the questions worth asking you first, and the mistakes to avoid.
“Every hospital in the city.” “Every dealership in the state.” “Every gym within the metro.” It is a common ask and the usual answer is disappointing, because the aggregator directories that look like they have the list are stale, partial, and thin on the fields that matter. This blueprint builds the list from the businesses’ own sites instead. Their pages are where the current address, the real opening hours and the actual service list live, because the business updates them and nobody else does.

​
Start with an honest base list

Before the pipeline, decide where the universe of businesses comes from. Map products are not scrapeable. Their terms forbid it and the results live inside an interactive pane rather than on a page, so anyone offering you a map scrape is selling you a liability rather than a dataset. Use the licensed route: a places API with a category and a radius, or OpenStreetMap through Overpass for an open licensed alternative. Both give you names and approximate locations, and both stop well short of a usable record. That is the split worth understanding. A licensed places source gives you the spine. This pipeline gives you the record.

​
The pipeline

​
1. Find each business’s own site

search reaches past the aggregators to the pages the businesses publish themselves.
Search a neighbourhood at a time rather than a whole city. The results are ranked, so a broad query returns the same handful of national aggregators everywhere, while a narrow one surfaces the local operators you actually wanted. Add "scrapeResults": true when you want each result’s content in the same call. That bills 1 credit per result on top of the search, so leave it off while you are still deciding which results are worth keeping.

​
2. Enumerate every branch of a multi-location business

Most categories are dominated by a handful of chains with dozens or hundreds of locations each. Search will show you one branch. map gives you all of them.
The search filter is a substring match against the URLs, so a city name or a path segment narrows a national site down to one region without fetching anything. This is the step where coverage actually gets built: one call, one credit, and the long tail of branches no directory bothered to list. Raise limit when the result comes back exactly at the number you asked for, because that means you hit your ceiling rather than the end of the list.

​
3. Turn each page into a record

One schema, written once, applied across operators whose sites have nothing structurally in common.
The records come back shaped the way you asked:
homeCollection is the field to pay attention to. It is meaningless outside diagnostics, and that is the point: the last field in the schema is usually the one specific to your category, and it is the reason to write a schema rather than buy a list. A dealership directory wants brandsSold, a gym wants openTwentyFourHours, a clinic wants the specialisms. No aggregator carries any of them, because no aggregator knows which vertical you are in. urls takes a list, so a batch of branches is one call rather than one call per branch.

​
Deduplicate before you extract, not after

Extraction is the priced step, so paying for it twice on the same location is the easiest money to waste here. The same branch is routinely reachable from several URLs: a city landing page, a locality page, a legacy path kept alive for search. Normalise the list first, meaning drop tracking parameters, settle on one trailing slash convention, and collapse obvious aliases. Then, after extraction, deduplicate again on the record itself, because two genuinely different URLs can describe one physical location. A phone number plus a postal code is a stronger key than a name. Names carry franchise suffixes, punctuation and spelling drift that no normalisation fully fixes.

​
Keeping it current

Local business data goes stale faster than almost anything else. Branches open, move, change hours and quietly close. Re-run map on a schedule, since it is 1 credit and the diff is most of the value. A URL that appeared is a new branch. A URL that disappeared is a closure or a redesign, and either way it is the signal you wanted. Re-extract selectively. Hours and services change; addresses rarely do. Extract new URLs always and cycle through existing ones slowly, rather than re-running the whole city every week. Mark what left rather than deleting it. A directory that silently drops a closed location cannot tell you it closed, and that is often the question being asked.

​
What it costs

Search and map are 1 credit each. Extract is 5 credits per URL, because it is model-backed work rather than a fetch. A worked example: one operator, 200 branches in a city, full records. That is 1 credit to map and 1,000 to extract. Repeat it for the three or four operators that cover most of a category and you have the city, with fields no directory sells. For thousands of pages, use batch rather than looping extract calls. It takes the same options, returns a job id, and calls a webhook when it finishes, so nothing sits waiting on a response. Failed calls never cost credits, so the cheapest way to size any of this is to run step one and look at what comes back.

​
What to watch for

Validate URLs before you spend on them. A URL that does not resolve returns a record of nulls rather than an error, which reads like a failed extraction when it is really a bad input. Confirm the page exists before extracting it. Not every business has a website, and in some categories that is most of them. Those businesses need a different route entirely, usually a phone call or a licensed dataset, and the pipeline should record them as “no source” rather than as failures. Watch the fill rate per field. Extraction keeps returning valid JSON after a site redesign, with the field you cared about silently null on every page. Track the share of records where each field came back non-null, and alert when a field that was consistently full drops sharply. Ask for fewer fields than you think you need. Three fields that are right on every page beat twelve that are right on half. Public pages only. Anything behind a login or a booking flow is out of scope, and a record built from what a business chose to publish is the one you can stand behind.