Why We Built a Rust Browser Engine

AI agents need to browse the web. To do this effectively, they require a way to render JavaScript, handle complex DOM structures, and extract clean data without the overhead of a full desktop browser. Most developers solve this by wrapping headless Chrome, but the resource costs of doing so are unsustainable at scale. We decided to build a custom rust browser engine to eliminate these inefficiencies.

What is a rust browser engine?

A browser engine is the core software component that transforms HTML, CSS, and JavaScript into a rendered page. While a full browser (like Chrome or Firefox) includes a user interface, tab management, and extension support, the engine is the "headless" part that does the actual work of parsing and rendering.

Building a rust browser engine means implementing these components using the Rust programming language. Rust is a systems language that provides memory safety without a garbage collector (GC). In the context of a browser engine, this allows for precise control over how memory is allocated and freed, which is critical when processing thousands of concurrent web pages.

Most existing scraping tools are "wrappers." They use a language like Node.js or Python to send commands to a separate Chrome process via the Chrome DevTools Protocol (CDP). A native Rust engine, by contrast, integrates the parsing and execution logic directly into the application binary.

Why a custom engine matters for AI agents

The current industry standard—headless Chrome—was designed for humans, not for AI agents. This design creates three primary bottlenecks: memory consumption, startup latency, and infrastructure complexity.

The Memory Problem

A single headless Chrome instance typically consumes between 200MB and 500MB of RAM. For a developer running a RAG (Retrieval-Augmented Generation) pipeline that needs to scrape hundreds of pages simultaneously, this becomes a massive cost center.

By building in Rust, we reduced the memory footprint per session to approximately 2MB. This is a 100x reduction in RAM usage. On a modest $5/month server, you can run 1,000 concurrent sessions with a Rust engine, whereas you might only manage 5 to 10 Chrome sessions before the system runs out of memory.

The Cold Start Problem

In a serverless or auto-scaling environment, the time it takes for a browser to start (the "cold start") is a critical metric. Spawning a Chrome process is slow; it involves initializing a heavy executable and allocating significant memory. Cold starts for Chrome-based solutions typically range from 500ms to 2,000ms.

Our rust browser engine achieves a median cold start time of 0.19ms. This sub-millisecond startup allows the API to respond almost instantly, which is essential for autonomous agents that need to perform a "search-read-act" loop in real-time.

Infrastructure Complexity

Chrome is a massive dependency. Deploying it requires specific shared libraries, large Docker images (often 1GB to 2GB), and complex process management to prevent "zombie" Chrome processes from eating CPU cycles.

A Rust-based approach compiles everything into a single binary. This results in a compressed Docker image of only 12MB. It simplifies deployment, reduces the attack surface for security vulnerabilities, and eliminates the need for external browser binaries.

How a rust browser engine works

Building a browser engine from scratch is a massive undertaking. Instead of reinventing every wheel, we utilized high-performance Rust libraries to build a modular pipeline.

HTML Parsing with html5ever

To handle the messiness of the modern web, we use html5ever, the same HTML5 parsing library used by the Servo project and Firefox. It is designed to be highly compliant with the HTML specification, ensuring that even malformed HTML is parsed into a consistent DOM (Document Object Model) tree.

The parsing process happens in microseconds. For a typical 12KB page, our engine parses the HTML in 134μs. Because Rust allows for zero-cost abstractions, we can navigate this DOM tree and query CSS selectors in as little as 496ns.

JavaScript Execution with Boa

JavaScript rendering is the hardest part of web scraping. Most sites use React, Vue, or Next.js, meaning a simple HTTP request returns an empty shell. You need a JS engine to execute the scripts and populate the page.

We integrated Boa, a JavaScript engine written entirely in Rust. Unlike V8 (the engine inside Chrome), which is written in C++, Boa allows the entire execution stack to remain within the Rust safety model. This prevents the memory leaks and segmentation faults common in C++ based engines.

The Async Runtime

To handle thousands of concurrent requests, we use the Tokio async runtime. This allows the engine to handle I/O operations—like fetching a page from a remote server—without blocking the execution thread. By combining Tokio with the Axum web framework, we can maintain a p95 API latency of 47ms.

The Rust browser engine in practice

To understand the difference in a real-world scenario, consider the workflow of an AI agent attempting to read a documentation page to answer a user's question.

The Chrome-based Workflow:

1. The agent triggers a scrape request.
2. The system spawns a headless Chrome process (500ms+ latency).
3. Chrome allocates 300MB of RAM.
4. Chrome loads the page, renders the full CSS layout, and executes JS.
5. The wrapper extracts the HTML and converts it to markdown.
6. The Chrome process is killed to reclaim memory.

The Ilmenite Workflow:

1. The agent triggers a scrape request via Ilmenite's API.
2. The Rust engine starts in 0.19ms.
3. The engine allocates ~2MB of RAM.
4. The engine parses HTML and executes necessary JS via Boa.
5. The engine generates clean markdown in 3.84μs.
6. The memory is immediately freed by Rust's ownership model.

This efficiency is why we can process 2.4 billion pages every 90 days while maintaining 99.98% uptime.

Honest Trade-offs: Rust vs. Chrome

While a custom Rust engine is superior for speed and cost, it is not a complete replacement for Chrome in every single use case. Transparency is important for developers choosing their stack.

JavaScript Performance

V8 (Chrome's engine) is arguably the most optimized piece of software on earth. It uses Just-In-Time (JIT) compilation to execute JavaScript at near-native speeds. Boa, while fast and memory-safe, does not yet match V8's raw execution speed for extremely complex, compute-heavy Single Page Applications (SPAs).

If a website relies on massive, complex JS calculations to render its content, Boa may struggle. To solve this, we implemented a fallback mechanism. When the engine detects a page that requires V8-level performance, it automatically routes the request to a Chrome instance. This gives users the speed of Rust for 90% of the web and the compatibility of Chrome for the remaining 10%.

CSS Rendering

Chrome renders a pixel-perfect visual representation of a page. Our engine focuses on the DOM and the data. Because we are optimizing for AI agents—which read markdown, not pixels—we do not perform full CSS layout rendering. If you need a pixel-perfect screenshot for a visual regression test, a full browser is still the right tool. For extracting data to feed into an LLM, the overhead of CSS rendering is wasted effort.

Tools and Resources

If you are interested in building with Rust or exploring browser internals, there are several projects that lay the groundwork for this technology:

• Servo: A research project developing a parallel browser engine in Rust. It is the inspiration for much of the modern Rust web ecosystem.
• Boa: The Rust-native JavaScript engine. It is an excellent resource for understanding how JS bytecode is executed without a C++ runtime.
• html5ever: The gold standard for HTML5 parsing in the Rust ecosystem.
• Tokio: The essential async runtime for any high-performance Rust networking tool.

For developers who want the benefits of a rust browser engine without the effort of building and maintaining the infrastructure, you can use Ilmenite. We provide the managed API, the SDKs, and the scale, allowing you to focus on building your AI agent rather than managing headless browser processes.

You can explore our documentation to see how to integrate this performance into your RAG pipeline or AI agent.

Ready to stop managing Chrome instances? Start for free and experience sub-millisecond startup times.