> ## Documentation Index > Fetch the complete documentation index at: https://docs.hydrafetch.com/llms.txt > Use this file to discover all available pages before exploring further. # Start a crawl > Discover and scrape a whole site as one asynchronous job. Returns a crawl id immediately; poll it or register a webhook. One credit per page scraped. ## OpenAPI ````yaml https://api.hydrafetch.com/openapi.json post /v1/web/crawl openapi: 3.0.0 info: title: Hydrafetch API description: >- Hydrafetch turns any URL into clean, LLM-ready data through one API. Give us a link and get back Markdown, the page's own structured data, extracted JSON, links, a summary, or a screenshot. Point us at a whole site and get every page. Ask a question and get answers with per-field confidence and the passage each value came from. You describe the outcome you want — the pipeline decides how to get it. ## Authentication Every request is authenticated with your API key in the `X-API-Key` header. Keys are scoped to a workspace and carry its credit balance. ## Credits Calls are billed in credits and charged only on success. A standard scrape is one credit; richer formats and the extraction tier cost more. Each response reports what it consumed. ## Conventions All timestamps are UTC ISO 8601. Long-running jobs (crawl, batch) return a job id you poll, or a webhook you register. Errors return a structured body with a stable `code` and an HTTP status. version: '1.0' contact: name: Hydrafetch url: https://hydrafetch.com email: team@hydrafetch.com servers: - url: https://api.hydrafetch.com description: Production security: - apiKey: [] tags: - name: Web Scraping description: >- Turn URLs into clean, LLM-ready content: scrape a single page, crawl or batch-scrape a whole site, map its URLs, search the web, and capture screenshots or images. - name: Web Extraction description: >- Pull schema-shaped JSON out of one or many pages, with optional per-field confidence and the source passage behind each value. paths: /v1/web/crawl: post: tags: - Web Scraping summary: Start a crawl description: >- Discover and scrape a whole site as one asynchronous job. Returns a crawl id immediately; poll it or register a webhook. One credit per page scraped. operationId: startCrawl parameters: [] requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/CrawlRequestDto' responses: '201': description: The crawl was accepted. content: application/json: schema: $ref: '#/components/schemas/CrawlAcceptedDto' components: schemas: CrawlRequestDto: type: object properties: url: type: string example: https://example.com description: The site to start from. Must be http(s). limit: type: number minimum: 1 maximum: 5000 example: 100 description: Maximum number of pages to scrape. maxDepth: type: number minimum: 0 maximum: 10 example: 2 description: How many links deep from the starting page to follow. includePaths: maxItems: 50 example: - ^/blog/.* description: Only follow URLs whose path matches every one of these patterns. type: array items: type: string excludePaths: maxItems: 50 example: - ^/tag/.* description: Skip URLs whose path matches any of these patterns. type: array items: type: string allowSubdomains: type: boolean example: false description: Also follow links into subdomains of the starting site. Default off. allowExternalLinks: type: boolean example: false description: Also follow links that lead off the starting site. Default off. ignoreQueryParameters: type: boolean example: false description: >- Treat URLs that differ only by query string as the same page. Default off. sitemap: type: string enum: - skip - include example: include description: >- Whether to seed discovery from the site's published page list. Default includes it. webhook: description: >- Register a callback to be notified as the crawl progresses instead of polling. allOf: - $ref: '#/components/schemas/WebhookDto' scrapeOptions: description: How to scrape each page. Same options as a single scrape. allOf: - $ref: '#/components/schemas/ScrapeOptionsDto' required: - url CrawlAcceptedDto: type: object properties: crawlId: type: string example: 019f3c09-6fae-740f-9257-10c2b6af7f43 description: Poll this crawl id for progress and results. status: type: string enum: - queued example: queued description: The job has been accepted. required: - crawlId - status WebhookDto: type: object properties: url: type: string example: https://your-app.com/hooks/hydrafetch description: >- The URL to POST events to. Must be a public http(s) URL — private, loopback, and link-local addresses are rejected. events: type: array example: - page - completed description: >- Which events to receive. `page` fires as each page finishes; `completed` fires once when the job ends. Defaults to both. items: type: string enum: - page - completed headers: type: object additionalProperties: type: string example: Authorization: Bearer your-token description: Extra headers to send with each callback, e.g. for authentication. secret: type: string example: whsec_a_long_random_string minLength: 16 maxLength: 256 description: >- A shared secret. When set, each callback carries an X-Hydrafetch-Signature header you can verify to prove the request came from us and is not a replay. required: - url ScrapeOptionsDto: type: object properties: formats: type: array example: - markdown description: What to return per page. Omit for Markdown only. items: type: string enum: - markdown - html - rawHtml - links - structured onlyMainContent: type: boolean description: Return only the main content, dropping boilerplate. Default true. includeTags: maxItems: 50 description: CSS selectors to keep. type: array items: type: string excludeTags: maxItems: 50 description: CSS selectors to strip before extraction. type: array items: type: string removeBase64Images: type: boolean description: Strip inline base64 images. Default true. blockAds: type: boolean description: Remove common ad and tracking elements. Default true. includeLinks: type: boolean description: >- Keep inline links in the markdown. Default false — dropping them is what keeps our output dense. Turn on for reference, API-docs and code pages where cross-references are content. renderJs: type: boolean description: Force full page rendering for JavaScript-heavy pages. waitFor: type: number minimum: 0 maximum: 30000 description: Extra milliseconds to let the page settle before capture. timeout: type: number minimum: 1000 maximum: 120000 description: Per-page time budget, in milliseconds. location: $ref: '#/components/schemas/OptLocationDto' headers: type: object additionalProperties: type: string description: Extra request headers to send when fetching each page. preferStructure: type: boolean description: Preserve document structure over prose density. Default off. maxAge: type: number minimum: 0 maximum: 604800000 description: Serve a page from cache if younger than this many milliseconds. OptLocationDto: type: object properties: country: type: string example: us description: ISO 3166 alpha-2 country to fetch as if from. languages: example: - en-US - en description: Preferred content languages, most-preferred first. type: array items: type: string securitySchemes: apiKey: type: apiKey in: header name: X-API-Key ````