Map

Map enumerates a site’s URLs quickly, without scraping any of them. It returns synchronously and costs one credit — the fastest way to see what pages a site has before committing to a crawl.

When to use

You want to preview a Crawl’s scope before running it.
You need a list of a site’s URLs to filter down and feed into a Batch.
You want to find pages matching a term (e.g. every URL containing pricing).

Map returns URLs only — no page content. To get content, pass the URLs to Scrape, Batch, or Crawl.

Example request

Send a POST to /v1/web/map. Narrow the results with search, limit, and discovery options.

curl -X POST https://api.hydrafetch.com/v1/web/map \
  -H "X-API-Key: hf_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "search": "pricing",
    "limit": 1000
  }'

const res = await fetch("https://api.hydrafetch.com/v1/web/map", {
  method: "POST",
  headers: {
    "X-API-Key": "hf_your_key_here",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    url: "https://example.com",
    search: "pricing",
    limit: 1000,
  }),
});
const { data } = await res.json();
console.log(data.links);

import requests

res = requests.post(
    "https://api.hydrafetch.com/v1/web/map",
    headers={"X-API-Key": "hf_your_key_here"},
    json={"url": "https://example.com", "search": "pricing", "limit": 1000},
)
print(res.json()["data"]["links"])

Example response

{
  "data": {
    "url": "https://example.com",
    "links": [
      "https://example.com/",
      "https://example.com/pricing"
    ],
    "count": 2
  }
}

Request options

string

required

The site whose URLs you want to enumerate. Must be http(s).

boolean

default:"false"

Also include same-site links found on the starting page, not just the site’s page list.

number

Maximum number of URLs to return. 1–5000. Results are ordered by how recently each page changed, so a limit gives you the most recently updated URLs rather than an even sample of the site. On a site where one kind of page changes constantly, such as a shop’s product pages, a small limit can return that kind and nothing else. sitemapInclude is how you ask for the rest.

string

Keep only discovered URLs containing this term (e.g. "pricing"). Up to 200 characters.

string

Whether to use the site’s published page list: skip it, include it alongside discovered links, or use it only.

string

On sites that publish several page lists, only read the ones whose address contains one of these comma-separated terms. For example "pages" on a shop to reach its policy and information pages instead of its products.

string

Skip page lists whose address contains any of these comma-separated terms. Compose it with sitemapInclude, for example include "post" and exclude "tag,category,author" to avoid tag and category archives.

string

default:"newest"

Sort by how recently each page changed: newest first, or oldest. URLs with no recorded date sort last either way.

boolean

default:"false"

Also include URLs on subdomains of the site.

boolean

default:"false"

Treat URLs that differ only by query string as one.

Response fields

object

Show MapResult

string

The site you requested.

string[]

The site’s discovered URLs.

number

How many URLs were returned.

Map is one credit regardless of how many URLs come back, and it returns synchronously. It is the cheapest way to size up a site before a crawl or batch.

Next steps

Map API reference

Full request and response schema with a live playground.

Crawl a whole site

Turn a mapped scope into scraped pages.

Scrape a list of URLs

Feed mapped URLs into a batch job.

Scrape one URL

Scrape any single discovered URL.

Get started

Core concepts

Web Scraping

Web Extraction

When to use

Example request

Example response

Request options

Response fields

Next steps

Map API reference

Crawl a whole site

Scrape a list of URLs

Scrape one URL

​When to use

​Example request

​Example response

​Request options

​Response fields

​Next steps

Map API reference

Crawl a whole site

Scrape a list of URLs

Scrape one URL

When to use

Example request

Example response

Request options

Response fields

Next steps