Exa Improves Embedding Throughput by 47% Using AWS AI Hardware Optimizations
Exa optimized its semantic embedding pipeline, achieving a 47% improvement in performance and cost per million documents by tuning for AWS Inferentia2 and Trainium2 hardware. This update is relevant for developers responsible for large-scale embedding pipelines or interested in cost-efficient AI infrastructure.
What changed?
Exa, a web-scale neural search platform, optimized its embedding pipeline for higher throughput and lower cost by targeting AWS Inferentia2 and Trainium2 chips. The team adopted a mix of weight layout changes, enabled mixed precision (autocast), and introduced custom fused kernels, particularly for SwiGLU MLP layers. These changes delivered a 47% improvement over the previous baseline in embedding throughput and cost per million documents embedded.

Why does it matter to an everyday developer?
Developers working with large-scale vector databases or semantic search have direct hardware and cost constraints when embedding billions of documents. Exa's optimizations show how targeted inference tuning on specialized AI accelerators can provide nearly 1.5× more throughput and substantial cost savings. This is especially actionable for teams deploying models at web scale or for those considering AWS AI hardware for inference tasks. The approach demonstrates that custom kernel and memory layout work, coupled with hardware-specific compiler optimizations, can have multiplicative impact without needing model architecture changes.
What can the developer do now?
- If you operate large embedding or inference workloads, evaluate AWS Inferentia2 and Trainium2 instance types for cost-performance benefits.
- Profile current pipelines for memory stalls or inefficient FLOPS usage, especially if using generic ML kernels.
- Consider custom kernel development using frameworks like AWS Neuron Kernel Interface (NKI) to address hardware-specific bottlenecks.
