Skip to main content

Troubleshooting

Common errors you may hit, what they mean, and how to fix them.

Authentication​

401 Unauthorized — Invalid or missing API key​

The fail-closed policy is at work:

  • PRY_API_KEY set → every request needs Authorization: Bearer <key>. Add the header, or generate a key with python -c "import secrets; print(secrets.token_urlsafe(48))".
  • PRY_API_KEY unset → only loopback (127.0.0.1/::1) is allowed. If you're calling from another host, either set a key or run the request from the host machine.

Proxy/Tor config endpoints (/v1/proxy/configure, POST /v1/config, /v1/config/profile/tor) are guarded in addition to the middleware — remote clients must present the key.

Rate limiting​

429 Too Many Requests — Rate limit exceeded​

Default limit is 120 requests/minute per IP (PRY_RATE_LIMIT_RPM). The response includes Retry-After and x-ratelimit-* headers.

Fixes:

  • Back off: respect Retry-After.
  • Raise the limit: set PRY_RATE_LIMIT_RPM higher (and restart).
  • Distribute load: batch work through /v1/batch (up to 50 URLs per call).

Cloudflare / blocking​

Scrape returns HTML with a Cloudflare challenge​

Check which tier succeeded:

curl -X POST http://localhost:8005/v1/ultimate-scrape \
  -H "Content-Type: application/json" \
  -d '{"url": "https://challenged.example.com"}'
# look at method_used in the response
  • If method_used is direct, the site is likely fine without bypass.
  • If challenges persist, verify FlareSolverr is healthy: docker compose ps — the pry service waits for flaresolverr to be healthy before starting.
  • Use POST /v1/detect-block to identify the vendor (Cloudflare, DataDome, …) and confidence.
  • Enable the stealth stack (PRY_STEALTH_ENABLED=true) and residential proxies (WEBSHARE_PROXY_LIST) for aggressive WAFs.

POST /v1/detect-block — what does the response mean?​

It returns the detected protection tier, vendor, and confidence. Use it to decide whether to invest in proxies, browser tiers, or a different source for the data.

Ports & connectivity​

Connection refused on http://localhost:8005​

  • Docker publishes host 8005 → container 8002. From the host, use 8005.
  • Port 8002 is only reachable if you docker exec into the container.
  • Check the service is up: docker compose ps and curl -fsS http://localhost:8005/health.

Health check fails in Docker​

The container healthcheck probes http://localhost:8002/health (container port). If the app crashes at startup (bad .env, missing secrets), the container never becomes healthy. Check logs:

docker logs pry --tail 100 -f

Tor​

Tor tier errors with missing aiohttp-socks​

The Tor routing tier is documented but currently disabled in some installs — aiohttp-socks is missing from dependencies. Either install it (pip install aiohttp-socks), enable the tor compose profile, or let the fallback chain skip the Tor tier (it continues to the archive tiers).

x402 payments​

402 Payment Required on every call​

x402 gating is enabled (PRY_X402_ENABLED=true) and you haven't paid. Either:

  1. Complete the flow: pay the wallet from the PAYMENT-REQUIRED header, then POST /v1/x402/pay with the tx_hash, and send the returned payment_id as X-Payment-Id.
  2. Disable gating (PRY_X402_ENABLED=false) if you didn't intend to charge.

underpayment: client supplied X USD, server requires Y USD​

The server enforces the authoritative price (validate_client_amount). Send at least the listed price from GET /v1/x402/pricing for the operation.

Payment verifies but the endpoint still 402s​

Check PRY_X402_PAYMENT_TTL (default 3600s) — the payment_id expires and must be re-paid. Also confirm you're sending the X-Payment-Id header (case-insensitive) on the paid call.

PRY_X402_OFFLINE=true is ignored​

Offline (dev-only) mode requires DEBUG=true or PRY_DEBUG=true as a second gate. It's intentional — never run offline verification in production.

Migrations​

Container won't start — Alembic error​

The entrypoint runs alembic stamp head (idempotent), falling back to alembic upgrade head on failure. If a migration is genuinely broken:

docker exec pry alembic current
docker exec pry alembic upgrade head

Use PRY_SKIP_MIGRATIONS=1 only for read-only debug runs.

Common response-shape gotchas​

422 Validation Error​

Pydantic rejected the body. Check the documented fields (e.g. url is required on /v1/scrape, steps is required on /v1/automate, schema is required on /v1/extract/css). The error response includes field-level details.

Error format​

All errors are RFC 7807 Problem Details with type, title, status, detail, request_id, instance, timestamp, code. Match on code or status, and log request_id for correlation — see API Overview → Error format.

Still stuck?​