Exa Achieves 50% Throughput Boost for Web Indexing on AWS AI Chips

Follow us on Google News

Exa, a company building a comprehensive web index for semantic search and agent grounding, has significantly enhanced its embedding infrastructure, achieving a 50% increase in throughput for indexing billions of documents. This optimization was realized by leveraging AWS Inferentia2 and Trainium2 AI chips, with a particular focus on improving the cost-effectiveness and flexibility of inference workloads.

The Chenab Times has learned that Exa’s core technology is an in-house vector database designed to ingest and embed vast amounts of high-quality web data. The unit economics of this embedding process are critical for the company’s operations at the scale of tens to hundreds of billions of documents, where even small per-document cost improvements yield substantial savings.

The optimization journey involved an intensive focus on AWS Neuron hardware, specifically Inferentia2 and Trainium2 chips. Exa’s infrastructure runs on an asynchronous pipeline that demands high volume and speed to ensure recent data is served quickly. Every structured web document deemed high quality must be embedded before it can be made available through the index.

During a recent backfill process to re-embed its existing corpus with a superior document model, Exa experienced a doubling of its embedding load. This critical juncture prompted a re-evaluation of its hardware choices and optimization strategies. The company utilized a configuration of Inferentia2 and Trainium2 NeuronCores, managing workloads through Kubernetes pods and ensuring container homogeneity across its embedding fleet.

Exa’s initial implementation on Inferentia2 was a standard data-parallel FP16 model. The subsequent optimization efforts, guided by the Neuron SDK and AWS Trainium team support, concentrated on maximizing throughput, measured in documents per second. These optimizations included weight layout adjustments to align with the model’s scheduling and reduce unnecessary transposes, as well as enabling autocast to allow the compiler to downcast matrix multiplications to lower precision while maintaining accuracy.

Further performance gains were realized through the use of the Neuron Kernel Interface (NKI) library, which provides optimized kernels for common model layers. A drop-in attention kernel offered an initial performance bump. The primary bottleneck identified was memory spill, where excessive data movement between high-bandwidth memory (HBM) and the Tensor Engine hindered full utilization. To address this, Exa developed a custom weight-stationary fused SwiGLU MLP kernel, written in a combination of higher-level NKI language for clarity and lower-level NKI ISA for specific tensor operations, effectively eliminating HBM stalls.

Even after addressing memory issues, a performance gap persisted between single and dual-core utilization, indicating contention. By rewriting the attention kernel to precisely match the model’s specific head architecture shapes, Exa reduced the number of floating-point operations (FLOPS) and eliminated padding waste. Folding attention operations into single matrix multiplications further reduced HBM round trips and unlocked significant speedups. Cumulatively, these optimizations resulted in a 47% improvement in performance and cost per 1 million documents embedded compared to the initial baseline.

The optimization process was underpinned by an agentic methodology. Using Flyte, an ML ops workflow platform, agents autonomously compiled Neuron executables, verified correctness against a baseline dataset (requiring 0.999 cosine similarity), profiled workloads, and conducted saturation benchmarks. This system allowed lightweight agents to handle tasks like grid searching compiler parameters, while reasoning agents identified bottlenecks and code-generating agents developed custom kernels.

The benefits of this flexible compute approach extend beyond a single hardware type or region. Exa schedules compute across multiple AWS Regions using a single Kubernetes control plane, comparing performance per dollar across different instance types and dynamically reallocating workloads. This strategy allows Exa to leverage the cheapest available nodes, adapt to shifting embedding demands, and absorb unused reserved compute. The optimizations developed for Inferentia2 were directly transferable to Trainium2, demonstrating the value of hardware-agnostic optimization efforts.

Ultimately, Exa’s focus on extracting maximum cost-performance from a flexible pool of hardware, amplified by optimizing Neuron hardware with NKI, has delivered substantial long-term cost savings for its web-scale indexing operations.

logo

The Chenab Times News Desk

Independent Journalism • Public Interest

Stand With Independent Journalism

Independent journalism survives on public trust and public support. Your contribution helps The Chenab Times continue reporting, questioning power and bringing unheard voices to the forefront.

Choose Your Monthly Support

Supporter

₹99 per month

A small contribution. A meaningful difference.

SUPPORT WITH ₹99

Champion

₹499 per month

Strengthen our public-interest journalism.

SUPPORT WITH ₹499

Guardian

₹999 per month

Be a strong pillar of independent journalism.

SUPPORT WITH ₹999

Prefer a One-Time Contribution?

Every contribution counts. Support our work with an amount of your choice.

Secure payments via Razorpay
Independent. Reader-supported. Committed to public interest.
Recurring payment terms and cancellation are subject to the selected payment plan.