Scrape

Fetch one URL, render it, and return clean output in the format your agent or pipeline wants. The most common endpoint — it's what most users call first.

Endpoint

POST https://api.ilmenite.dev/v1/scrape

Request body

• url (string, required) — absolute URL to fetch.
• format (string, optional) — one of markdown, html, json, text. Defaults to markdown.
• wait_for_selector (string, optional) — CSS selector to wait for before extracting.
• wait_for_ms (number, optional) — fixed delay after navigation, in milliseconds.
• timeout_secs (number, optional) — max fetch time in seconds.
• engine (string, optional) — chrome (default, full JS) or static (fast, no JS).
• only_main_content (boolean, optional) — strip nav, header, footer, aside, scripts, ads.
• include_tags / exclude_tags (string[], optional) — CSS selectors to keep or remove.
• proxy (string, optional) — your own proxy URL (BYOP), format http://user:pass@host:port.
• stealth (boolean, optional) — route through ilmenite's residential pool for IP reputation. Works on ~90–95% of sites with anti-bot. Cost: 5× base rate. Respects a caller-supplied proxy.
• unlock (boolean, optional) — route through ilmenite's full anti-bot solver (bypasses Cloudflare Turnstile, DataDome, PerimeterX). Cost: 15× base rate. Overrides proxy because the unlocker runs its own browser internally.

Example — basic

curl -X POST https://api.ilmenite.dev/v1/scrape \
  -H "Authorization: Bearer $ILMENITE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://news.ycombinator.com",
    "format": "markdown",
    "wait_for_selector": "tr.athing"
  }'

Example — stealth mode (5× cost, residential proxy)

Use stealth: truefor sites that block datacenter IPs but don't enforce behavioural anti-bot. Most e-commerce, news, and business sites with basic blocking fall here.

curl -X POST https://api.ilmenite.dev/v1/scrape \
  -H "Authorization: Bearer $ILMENITE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://some-blocked-site.com",
    "stealth": true
  }'

Example — unlock mode (15× cost, full anti-bot solve)

Use unlock: true only for sites behind full behavioural anti-bot (Cloudflare Turnstile, DataDome, PerimeterX, Kasada, Akamai Bot Manager). Charged only when the unlock succeeds — failed attempts are free.

curl -X POST https://api.ilmenite.dev/v1/scrape \
  -H "Authorization: Bearer $ILMENITE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://turnstile-protected-site.com",
    "unlock": true
  }'

Response

{
  "url": "https://news.ycombinator.com",
  "status": 200,
  "title": "Hacker News",
  "markdown": "# Hacker News\n\n1. ...",
  "links": ["https://ycombinator.com", "..."],
  "metadata": {
    "lang": "en",
    "description": "...",
    "published_at": null
  },
  "latency_ms": 412,
  "render_time_ms": 0.19
}

Errors

• 400 invalid_url — URL failed parsing or scheme is unsupported.
• 402 payment_required — insufficient balance for the requested tier. Top up at /dashboard/billing.
• 408 timeout — page did not finish within timeout_secs.
• 422 wait_for_not_found — selector never appeared.
• 502 upstream_error — origin returned 5xx or refused connection.
• 503 service_unavailable — requested stealth or unlock but the feature is not enabled on this deployment.