Map

Map enumerates a site’s URLs quickly, without scraping any of them. It returns synchronously and costs one credit — the fastest way to see what pages a site has before committing to a crawl.

When to use

You want to preview a Crawl’s scope before running it.
You need a list of a site’s URLs to filter down and feed into a Batch.
You want to find pages matching a term (e.g. every URL containing pricing).

Map returns URLs only — no page content. To get content, pass the URLs to Scrape, Batch, or Crawl.

Example request

Send a POST to /v1/web/map. Narrow the results with search, limit, and discovery options.

curl -X POST https://api.hydrafetch.com/v1/web/map \
  -H "X-API-Key: hf_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "search": "pricing",
    "limit": 1000
  }'

const res = await fetch("https://api.hydrafetch.com/v1/web/map", {
  method: "POST",
  headers: {
    "X-API-Key": "hf_your_key_here",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    url: "https://example.com",
    search: "pricing",
    limit: 1000,
  }),
});
const { data } = await res.json();
console.log(data.links);

import requests

res = requests.post(
    "https://api.hydrafetch.com/v1/web/map",
    headers={"X-API-Key": "hf_your_key_here"},
    json={"url": "https://example.com", "search": "pricing", "limit": 1000},
)
print(res.json()["data"]["links"])

Example response

{
  "data": {
    "url": "https://example.com",
    "links": [
      "https://example.com/",
      "https://example.com/pricing"
    ],
    "count": 2
  }
}

Request options

string

required

The site whose URLs you want to enumerate. Must be http(s).

boolean

default:"false"

Also include same-site links found on the starting page, not just the site’s page list.

number

Maximum number of URLs to return. 1–5000.

string

Keep only discovered URLs containing this term (e.g. "pricing"). Up to 200 characters.

string

Whether to use the site’s published page list: skip it, include it alongside discovered links, or use it only.

boolean

default:"false"

Also include URLs on subdomains of the site.

boolean

default:"false"

Treat URLs that differ only by query string as one.

Response fields

object

Show MapResult

string

The site you requested.

string[]

The site’s discovered URLs.

number

How many URLs were returned.

Map is one credit regardless of how many URLs come back, and it returns synchronously. It is the cheapest way to size up a site before a crawl or batch.

Next steps

Map API reference

Full request and response schema with a live playground.

Crawl a whole site

Turn a mapped scope into scraped pages.

Scrape a list of URLs

Feed mapped URLs into a batch job.

Scrape one URL

Scrape any single discovered URL.

Get started

Core concepts

Web Scraping

Web Extraction

When to use

Example request

Example response

Request options

Response fields

Next steps

Map API reference

Crawl a whole site

Scrape a list of URLs

Scrape one URL

​When to use

​Example request

​Example response

​Request options

​Response fields

​Next steps

Map API reference

Crawl a whole site

Scrape a list of URLs

Scrape one URL

When to use

Example request

Example response

Request options

Response fields

Next steps