Scrape, Crawl, and Extract API

Fetch one page, run a bounded crawl, or extract grounded answers and fields with Rankability's customer crawler API.

The crawler API exposes three related operations: fetch one page, crawl a bounded set of pages, or extract a grounded answer or structure from content. Use Site Auditor instead when you need technical audit findings rather than raw crawler output.

Choose an operation

Operation	Scope	Route	Execution
Scrape one page	scrape:run	POST /scrape	Synchronous
Start bounded crawl	crawl:run	POST /crawl	Asynchronous
Poll bounded crawl	crawl:run	GET /crawl/:jobId	Read/poll
Read daily page usage	scrape:run or crawl:run	GET /crawler/usage	Read-only
Extract an answer or fields	extract:run	POST /extract	Synchronous

Prefix paths with https://app.rankability.com/api/agent/v1.

Costs and cache behavior

• Scrape costs 10 credits after a successful non-cached page fetch. A failed fetch or reused cache hit costs 0.

• Crawl checks affordability for the requested maximum at creation, then charges 10 credits per successfully crawled page after completion.

• Extract costs 50 credits after a successful extraction. Too little content, a failed extraction, or a missing grounded answer does not automatically imply a fabricated result; inspect noContent, extractionFailed, and the returned mode-specific fields.

See API credits, rate limits, and errors for retry rules.

Scrape one page

curl -X POST https://app.rankability.com/api/agent/v1/scrape \
  -H "Authorization: Bearer rk_live_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/guide",
    "includeAeo": true,
    "query": "pricing and eligibility"
  }'

url is required. Optional fields are:

• includeAeo — include AI-answer-oriented extraction data.

• query — focus relevant-content extraction, up to 500 characters.

• maxAgeMinutes — reuse eligible cached data up to the requested age.

• pageActions — up to 12 click, scroll, wait, or acceptCookies actions. Use narrowly; selectors and timing are vulnerable to site changes.

The response includes the crawler jobId, pageId, page data, relevant content, cached, and creditsCharged. A returned page can contain its own errorMessage; check it before using the content.

Run a bounded crawl

curl -X POST https://app.rankability.com/api/agent/v1/crawl \
  -H "Authorization: Bearer rk_live_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/docs/",
    "maxPages": 25,
    "maxDepth": 2,
    "sameDomainOnly": true,
    "includeAeo": true
  }'

Current defaults are 25 pages, depth 2, same-domain only, and AEO extraction enabled. The hard limits are 200 pages and depth 5. Rankability clamps values above those ceilings.

The HTTP 202 response returns jobId, the initial job, and effective defaults. Poll:

curl https://app.rankability.com/api/agent/v1/crawl/JOB_ID \
  -H "Authorization: Bearer rk_live_YOUR_KEY"

Only the owning organization can read a job. The response includes job status, persisted pages, a fetch-source summary, and failed-result summary. Stop at a terminal job status and inspect failures before treating the page set as complete.

Extract grounded information

POST /extract accepts one source:

• url — fetch and analyze a public page.

• pageId — reuse a stored crawler page.

• content — analyze supplied text, up to 200,000 characters.

Choose one mode:

Mode	Required field	Result
question	question	Answer, answered, and supporting source highlights
highlights	query	Most relevant grounded passages
schema	fields	Caller-defined values plus missingFields

Example:

curl -X POST https://app.rankability.com/api/agent/v1/extract \
  -H "Authorization: Bearer rk_live_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/pricing",
    "mode": "schema",
    "fields": [
      { "name": "starting_price", "type": "number" },
      { "name": "free_trial", "type": "boolean" }
    ],
    "idempotencyKey": "example-pricing-v1"
  }'

For schema mode, missing evidence is returned as null and listed in missingFields. Do not replace nulls with guessed values.

Monitor the daily page quota

Before large scrape or crawl work, call GET /crawler/usage. The response and X-Quota-* headers report the current limit, used pages, remaining pages, and UTC window date.

A crawl checks its full requested maximum against the daily page quota when it is enqueued, even though billing uses successfully crawled pages. Reduce maxPages if the requested budget exceeds the remaining quota.

Common failures

• 400 invalid_input — invalid URL, fields, extraction mode, or required mode input.

• 402 insufficient_credits — the operation cannot pass its affordability gate.

• 404 not_found — a crawl or page is absent or belongs to another organization.

• 429 rate_limit_exceeded — the key-level request window is exhausted.

• 429 quota_exceeded — the daily crawler page budget is exhausted.

For complete workflow selection, see API use cases and integration patterns.