Scrape, Crawl, and Extract API

The crawler API exposes three related operations: fetch one page, crawl a bounded set of pages, or extract a grounded answer or structure from content. Use Site Auditor instead when you need technical audit findings rather than raw crawler output.

Choose an operation

Operation	Scope	Route	Execution
Scrape one page	scrape:run	POST /scrape	Synchronous
Start bounded crawl	crawl:run	POST /crawl	Asynchronous
Poll bounded crawl	crawl:run	GET /crawl/:jobId	Read/poll
Read daily page usage	scrape:run or crawl:run	GET /crawler/usage	Read-only
Extract an answer or fields	extract:run	POST /extract	Synchronous

Prefix paths with https://app.rankability.com/api/agent/v1.

Costs and cache behavior

• Scrape, Crawl, and Extract are included in full-platform pooled usage.
• Large bounded crawls can have a high on-demand impact; estimate scope before approval.
• Too little content, a failed extraction, or a missing grounded answer does not imply a fabricated result; inspect noContent, extractionFailed, and the returned mode-specific fields.

See API usage, rate limits, and errors for retry rules.

Scrape one page

curl -X POST https://app.rankability.com/api/agent/v1/scrape \
  -H "Authorization: Bearer rk_live_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/guide",
    "includeAeo": true,
    "query": "pricing and eligibility"
  }'

url is required. Optional fields are:

• includeAeo — include AI-answer-oriented extraction data.
• query — focus relevant-content extraction, up to 500 characters.
• maxAgeMinutes — reuse eligible cached data up to the requested age.
• pageActions — up to 12 click, scroll, wait, or acceptCookies actions. Use narrowly; selectors and timing are vulnerable to site changes.

The response includes the crawler jobId, pageId, page data, relevant content, cached, and pooled usageImpact. A returned page can contain its own errorMessage; check it before using the content.

Run a bounded crawl

curl -X POST https://app.rankability.com/api/agent/v1/crawl \
  -H "Authorization: Bearer rk_live_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/docs/",
    "maxPages": 25,
    "maxDepth": 2,
    "sameDomainOnly": true,
    "includeAeo": true
  }'

Current defaults are 25 pages, depth 2, same-domain only, and AEO extraction enabled. The hard limits are 200 pages and depth 5. Rankability clamps values above those ceilings.

The HTTP 202 response returns jobId, the initial job, and effective defaults. Poll:

curl https://app.rankability.com/api/agent/v1/crawl/JOB_ID \
  -H "Authorization: Bearer rk_live_YOUR_KEY"

Only the owning organization can read a job. The response includes job status, persisted pages, a fetch-source summary, and failed-result summary. Stop at a terminal job status and inspect failures before treating the page set as complete.

Extract grounded information

POST /extract accepts one source:

• url — fetch and analyze a public page.
• pageId — reuse a stored crawler page.
• content — analyze supplied text, up to 200,000 characters.

Choose one mode:

Mode	Required field	Result
question	question	Answer, answered, and supporting source highlights
highlights	query	Most relevant grounded passages
schema	fields	Caller-defined values plus missingFields

Example:

curl -X POST https://app.rankability.com/api/agent/v1/extract \
  -H "Authorization: Bearer rk_live_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/pricing",
    "mode": "schema",
    "fields": [
      { "name": "starting_price", "type": "number" },
      { "name": "free_trial", "type": "boolean" }
    ],
    "idempotencyKey": "example-pricing-v1"
  }'

For schema mode, missing evidence is returned as null and listed in missingFields. Do not replace nulls with guessed values.

Monitor the daily page quota

Before large scrape or crawl work, call GET /crawler/usage. The response and X-Quota-* headers report the current limit, used pages, remaining pages, and UTC window date.

A crawl checks its full requested maximum against the daily page quota when it is enqueued. Reduce maxPages if the requested budget exceeds the remaining quota.

Bounded polling and tenant isolation

Existing callers that omit view retain the original complete crawl response. New integrations should use an explicit bounded view:

• view=status returns only the job and progress fields.
• view=summary adds fetch-source and failure totals without page records.
• view=full adds a paginated page inventory. Use page_limit from 1–50, page_offset, and an optional comma-separated page_fields selection.

The default full-view field set is compact and excludes nested seo, aeo, scores, and object-storage paths. Select those fields only for the small page set that needs them.

Agent-created scrape and crawl records carry the owning organization. Freshness-cache lookup cannot reuse another organization’s record, crawl polling cannot reveal a foreign job, and pageId extraction accepts only a page belonging to the authenticated organization. These checks return not found rather than exposing whether a foreign record exists.

MCP equivalents

MCP clients can use get_crawler_usage, scrape_page, start_crawl, get_crawl, and extract_page_data. Before a run, call the current usage endpoint and estimate crawler_scrape, crawler_crawl, or crawler_extract; show the result, obtain approval, and send the same stable idempotency key on retries. Use status while a crawl is active, summary at completion, and full only for bounded page evidence.

Common failures

• 400 invalid_input — invalid URL, fields, extraction mode, or required mode input.
• 429 usage_limit_reached — a pooled on-demand window has reached its current limit.
• 404 not_found — a crawl or page is absent or belongs to another organization.
• 429 rate_limit_exceeded — the key-level request window is exhausted.
• 429 quota_exceeded — the daily crawler page budget is exhausted.

For complete workflow selection, see API use cases and integration patterns.