Scraping API

Core scraping endpoints. All examples assume a local instance at http://localhost:8005 (Docker) with a PRY_API_KEY set — replace the Authorization header with your key or drop it for loopback-only instances.

POST /v1/scrape

Scrape a single URL. Auto-bypasses Cloudflare by default and returns markdown (or JSON with a schema).

Request body (ScrapeRequest):

Field	Type	Default	Description
url	string	—	Required. URL to scrape
formats	array<string>	—	Output formats, e.g. ["markdown"], ["markdown","html"]
onlyMainContent	boolean	true	Strip navigation/ads and keep main content
timeout	integer	30	Timeout in seconds
bypassCloudflare	boolean	true	Attempt Cloudflare/WAF bypass
jsRender	boolean	false	Render JavaScript before extracting
jsonSchema	object	—	JSON schema for structured extraction

Example:

curl -X POST http://localhost:8005/v1/scrape \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <key>" \
  -d '{"url": "https://example.com", "bypassCloudflare": true, "formats": ["markdown"]}'

Response (200):

{
  "success": true,
  "data": {
    "url": "https://example.com",
    "markdown": "# Example Domain\n\nThis domain is for use in illustrative examples...",
    "metadata": {
      "title": "Example Domain",
      "status_code": 200,
      "method_used": "direct"
    }
  }
}

With jsonSchema, the response includes the extracted fields instead of (or in addition to) raw content.

POST /v1/crawl

Crawl multiple pages from a starting URL. Supports async webhooks.

Request body (CrawlRequest):

Field	Type	Default	Description
url	string	—	Required. Starting URL
maxPages	integer	10	Maximum pages to crawl
maxDepth	integer	2	Maximum link depth
scrapeOptions	object	—	Options forwarded to each page scrape
webhook	string	—	Async webhook URL for completion notifications

Example:

curl -X POST http://localhost:8005/v1/crawl \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <key>" \
  -d '{"url": "https://docs.com", "maxPages": 10, "maxDepth": 2, "timeout": 120}'

Response (200): an array of scraped pages, each with url, markdown/html content, and metadata.

POST /v1/batch

Scrape up to 50 URLs in parallel in one call.

Request body: { "urls": [...], "bypassCloudflare": true, "formats": ["markdown"] }

Field	Type	Default	Description
urls	array<string>	—	Required. URLs to scrape in parallel (max 50)
bypassCloudflare	boolean	true	Cloudflare bypass per URL
formats	array<string>	—	Output formats
timeout	query param	—	Optional query-string timeout

Example:

curl -X POST "http://localhost:8005/v1/batch?timeout=60" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <key>" \
  -d '{
    "urls": ["https://example1.com", "https://example2.com"],
    "bypassCloudflare": true,
    "formats": ["markdown"]
  }'

Response (200): an array of per-URL results (order matches the request).

note

Firecrawl charges extra for batch. Batch is included in Pry for free.

POST /v1/map

Discover URLs on a site (sitemap + link discovery).

Request body (MapRequest):

Field	Type	Default	Description
url	string	—	Required. Site to map
search	string	—	Filter discovered URLs by substring
ignoreSitemap	boolean	true	Skip sitemap discovery
limit	integer	50	Max URLs to return

Example:

curl -X POST http://localhost:8005/v1/map \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <key>" \
  -d '{"url": "https://docs.com", "limit": 50}'

Response (200): { "urls": ["https://docs.com/", "https://docs.com/guide", ...] }

POST /v1/ultimate-scrape

Scrape any URL using Pry's 10-tier anti-detection fallback system.

Automatically tries: direct → cloudscraper → FlareSolverr → undetected-chromedriver → Playwright → Googlebot → Archive.org → Google Cache (and more). Returns the first successful result with the method used.

Request body: { "url": "https://example.com" }

Example:

curl -X POST http://localhost:8005/v1/ultimate-scrape \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <key>" \
  -d '{"url": "https://example.com"}'

Response (200): scraped content plus method_used telling you which tier succeeded (e.g. flare, playwright, googlebot).

POST /v1/detect-block

Debug helper — detect what kind of anti-bot protection a site is using.

Request body: { "url": "https://example.com" }

Response (200): detection tier, vendor (Cloudflare / DataDome / …), and confidence score. Useful before tuning bypass settings.

Related scraping endpoints

Endpoint	Purpose
POST /v1/links	Link analysis for a page
POST /v1/batch-file	Batch scrape from a file + extraction template
POST /v1/parse	Parse PDF/DOCX/OCR/CSV/JSON — see Extraction API
POST /v1/capture/lazy	Lazy-load / infinite-scroll capture — see Automation API

Next steps

• Extraction API — structured data, parsing, shadow DOM
• Stealth & Anti-detection — how the fallback tiers work