> For the complete documentation index, see [llms.txt](https://help.rankability.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://help.rankability.com/api/api-crawler.md).

# Scrape, Crawl, and Extract API

The crawler API exposes three related operations: fetch one page, crawl a bounded set of pages, or extract a grounded answer or structure from content. Use [Site Auditor](/api/api-site-auditor.md) instead when you need technical audit findings rather than raw crawler output.

## Choose an operation

| Operation                   | Scope                       | Route                | Execution    |
| --------------------------- | --------------------------- | -------------------- | ------------ |
| Scrape one page             | `scrape:run`                | `POST /scrape`       | Synchronous  |
| Start bounded crawl         | `crawl:run`                 | `POST /crawl`        | Asynchronous |
| Poll bounded crawl          | `crawl:run`                 | `GET /crawl/:jobId`  | Read/poll    |
| Read daily page usage       | `scrape:run` or `crawl:run` | `GET /crawler/usage` | Read-only    |
| Extract an answer or fields | `extract:run`               | `POST /extract`      | Synchronous  |

Prefix paths with `https://app.rankability.com/api/agent/v1`.

## Costs and cache behavior

* Scrape, Crawl, and Extract are included in full-platform pooled usage.
* Large bounded crawls can have a high on-demand impact; estimate scope before approval.
* Too little content, a failed extraction, or a missing grounded answer does not imply a fabricated result; inspect `noContent`, `extractionFailed`, and the returned mode-specific fields.

See [API usage, rate limits, and errors](/api/api-credits-rate-limits-and-errors.md) for retry rules.

## Scrape one page

```bash
curl -X POST https://app.rankability.com/api/agent/v1/scrape \
  -H "Authorization: Bearer rk_live_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/guide",
    "includeAeo": true,
    "query": "pricing and eligibility"
  }'
```

`url` is required. Optional fields are:

* `includeAeo` — include AI-answer-oriented extraction data.
* `query` — focus relevant-content extraction, up to 500 characters.
* `maxAgeMinutes` — reuse eligible cached data up to the requested age.
* `pageActions` — up to 12 `click`, `scroll`, `wait`, or `acceptCookies` actions. Use narrowly; selectors and timing are vulnerable to site changes.

The response includes the crawler `jobId`, `pageId`, page data, relevant content, `cached`, and pooled `usageImpact`. A returned page can contain its own `errorMessage`; check it before using the content.

## Run a bounded crawl

```bash
curl -X POST https://app.rankability.com/api/agent/v1/crawl \
  -H "Authorization: Bearer rk_live_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/docs/",
    "maxPages": 25,
    "maxDepth": 2,
    "sameDomainOnly": true,
    "includeAeo": true
  }'
```

Current defaults are 25 pages, depth 2, same-domain only, and AEO extraction enabled. The hard limits are 200 pages and depth 5. Rankability clamps values above those ceilings.

The HTTP `202` response returns `jobId`, the initial job, and effective defaults. Poll:

```bash
curl https://app.rankability.com/api/agent/v1/crawl/JOB_ID \
  -H "Authorization: Bearer rk_live_YOUR_KEY"
```

Only the owning organization can read a job. The response includes job status, persisted pages, a fetch-source summary, and failed-result summary. Stop at a terminal job status and inspect failures before treating the page set as complete.

## Extract grounded information

`POST /extract` accepts one source:

* `url` — fetch and analyze a public page.
* `pageId` — reuse a stored crawler page.
* `content` — analyze supplied text, up to 200,000 characters.

Choose one mode:

| Mode         | Required field | Result                                               |
| ------------ | -------------- | ---------------------------------------------------- |
| `question`   | `question`     | Answer, `answered`, and supporting source highlights |
| `highlights` | `query`        | Most relevant grounded passages                      |
| `schema`     | `fields`       | Caller-defined values plus `missingFields`           |

Example:

```bash
curl -X POST https://app.rankability.com/api/agent/v1/extract \
  -H "Authorization: Bearer rk_live_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/pricing",
    "mode": "schema",
    "fields": [
      { "name": "starting_price", "type": "number" },
      { "name": "free_trial", "type": "boolean" }
    ],
    "idempotencyKey": "example-pricing-v1"
  }'
```

For schema mode, missing evidence is returned as null and listed in `missingFields`. Do not replace nulls with guessed values.

## Monitor the daily page quota

Before large scrape or crawl work, call `GET /crawler/usage`. The response and `X-Quota-*` headers report the current limit, used pages, remaining pages, and UTC window date.

A crawl checks its full requested maximum against the daily page quota when it is enqueued. Reduce `maxPages` if the requested budget exceeds the remaining quota.

## Bounded polling and tenant isolation

Existing callers that omit `view` retain the original complete crawl response. New integrations should use an explicit bounded view:

* `view=status` returns only the job and progress fields.
* `view=summary` adds fetch-source and failure totals without page records.
* `view=full` adds a paginated page inventory. Use `page_limit` from 1–50, `page_offset`, and an optional comma-separated `page_fields` selection.

The default full-view field set is compact and excludes nested `seo`, `aeo`, `scores`, and object-storage paths. Select those fields only for the small page set that needs them.

Agent-created scrape and crawl records carry the owning organization. Freshness-cache lookup cannot reuse another organization's record, crawl polling cannot reveal a foreign job, and `pageId` extraction accepts only a page belonging to the authenticated organization. These checks return not found rather than exposing whether a foreign record exists.

## MCP equivalents

MCP clients can use `get_crawler_usage`, `scrape_page`, `start_crawl`, `get_crawl`, and `extract_page_data`. Before a run, call the current usage endpoint and estimate `crawler_scrape`, `crawler_crawl`, or `crawler_extract`; show the result, obtain approval, and send the same stable idempotency key on retries. Use status while a crawl is active, summary at completion, and full only for bounded page evidence.

## Common failures

* `400 invalid_input` — invalid URL, fields, extraction mode, or required mode input.
* `429 usage_limit_reached` — a pooled on-demand window has reached its current limit.
* `404 not_found` — a crawl or page is absent or belongs to another organization.
* `429 rate_limit_exceeded` — the key-level request window is exhausted.
* `429 quota_exceeded` — the daily crawler page budget is exhausted.

For complete workflow selection, see [API use cases and integration patterns](/api/api-use-cases.md).