How to Run 1,000 Concurrent Scrapes for the Price of a Coffee

Running headless browsers at scale is usually a memory nightmare. If you use Chrome-based tools, each browser instance consumes between 200MB and 500MB of RAM. To handle 1,000 concurrent sessions, you would need roughly 200GB of memory, forcing you into expensive cloud instances — or into a browser-as-a-service provider that charges you by the browser-hour. This guide shows you how to achieve cheap web scraping at scale by using Ilmenite, a hosted scraping API built in Rust that bills a flat $0.001 per request with no browser-hour metering.

What we're building

We are going to hammer the Ilmenite API with 1,000 concurrent scraping requests from a single machine to verify that you can scale horizontally without paying per-minute browser rent. We will drive the load with a Python asyncio + httpx script and count successful responses, latency, and total cost. By the end of this tutorial, you will have a production-ready client pattern for flooding the hosted API with concurrent requests at $0.001 each.

Prerequisites

• Any machine that can run Python 3.11+ (a laptop is fine)
• An Ilmenite API key (get one via the sign up page — every account gets $5 in free balance every month)

Step 1: The Math of Cheap Web Scraping at Scale

Before writing the client, it is important to understand why this is possible. Most scraping APIs wrap Chromium and bill you by the browser-hour, so a page that finishes in 200ms still gets rounded up. Ilmenite is different: a dual-engine design routes simple HTTP GETs through a pure-Rust fast path (no browser at all) and reserves real Chromium only for JS-heavy pages. You pay a flat $0.001 per request either way.

Metric	Chrome-based scraping APIs	Ilmenite API
Billing	browser-hour metered	flat $0.001 per request
Static pages	always launch Chromium	pure-Rust fast path, no browser
JS-heavy pages	always launch Chromium	real Chromium (same API)
Free tier	usually none	$5/month balance, never expires

At $0.001 per request, 1,000 scrapes cost exactly $1. That is cheaper than the AWS data-transfer bill you'd pay to move the HTML off a browser-as-a-service node. Every account gets $5 free every month — so the first 5,000 scrapes every month are free.

Step 2: Configuring the Client

There is no server to provision. You are calling the hosted API directly at https://api.ilmenite.dev. All you need is your API key.

export ILMENITE_API_KEY="il_live_..."

Step 3: Testing Concurrency

To prove that we can achieve cheap web scraping at scale, we will use a Python script to send 1,000 concurrent requests to the hosted API. We will use asyncio and httpx for high-performance asynchronous I/O.

First, install the requirements:

pip install httpx

Create a script named load_test.py:

import asyncio
import httpx
import os
import time

# Configuration
API_URL = "https://api.ilmenite.dev/v1/scrape"
API_KEY = os.environ["ILMENITE_API_KEY"]
CONCURRENT_REQUESTS = 1000
TARGET_URL = "https://example.com"

async def scrape_page(client, request_id):
    payload = {
        "url": TARGET_URL,
        "format": "markdown"
    }
    headers = {
        "Authorization": f"Bearer {API_KEY}",
        "Content-Type": "application/json"
    }
    try:
        start = time.perf_counter()
        response = await client.post(API_URL, json=payload, headers=headers)
        end = time.perf_counter()
        if response.status_code == 200:
            return True, end - start
        return False, 0
    except Exception:
        return False, 0

async def main():
    limits = httpx.Limits(max_connections=CONCURRENT_REQUESTS, max_keepalive_connections=100)
    async with httpx.AsyncClient(limits=limits, timeout=30.0) as client:
        tasks = [scrape_page(client, i) for i in range(CONCURRENT_REQUESTS)]
        print(f"Launching {CONCURRENT_REQUESTS} concurrent requests...")

        start_time = time.perf_counter()
        results = await asyncio.gather(*tasks)
        end_time = time.perf_counter()

        successes = [r for r, t in results if r]
        total_time = end_time - start_time

        print(f"\n--- Results ---")
        print(f"Total Requests: {CONCURRENT_REQUESTS}")
        print(f"Successful: {len(successes)}")
        print(f"Total Time: {total_time:.2f}s")
        print(f"Requests per second: {len(successes)/total_time:.2f}")
        print(f"Total cost: ${len(successes) * 0.001:.3f}")

if __name__ == "__main__":
    asyncio.run(main())

Run the test:

python3 load_test.py

Step 4: Reading the Results

A successful run of 1,000 concurrent requests costs exactly $1.00 in API balance. Failed requests (status ≥ 400) are free — you are never charged for errors. Compare that to a browser-as-a-service provider that meters by the browser-hour and charges you for the time sessions stay open, regardless of whether they return data.

Concurrent request limits scale with your usage tier. The Free tier allows 2 concurrent, Starter allows 10, Growth allows 50, and Enterprise is unlimited. If you get HTTP 429s in the load test above, you are hitting your concurrency ceiling — top up your balance or contact us about Enterprise.

Full Working Implementation Summary

To recap, your setup consists of:

1. An API key from ilmenite.dev — $5 free every month, no credit card.
2. An asynchronous Python client using httpx to fan out requests in parallel.
3. A flat $0.001 per request cost model, with failed requests free.

The total cost for 1,000 scrapes is $1. The first 5,000 every month are free. Compare to the $200+ monthly spend required to host the RAM for 1,000 Chrome instances yourself, plus engineering time keeping the fleet alive.

Next Steps

Now that you have a high-throughput scraping client, you can integrate it into your larger AI pipeline. If you need to extract specific data points instead of raw markdown, explore the extract endpoint to return structured JSON based on a schema.

For those building autonomous agents, we recommend checking out our MCP integration, which allows Claude to call the Ilmenite API natively. For volume pricing, dedicated infrastructure, or SOC 2 compliance, see the pricing page or contact hello@ilmenite.dev.

Ready to scale your data collection? Start free and try the Ilmenite API today.