i14 has successfully developed and deployed its Laplace 3B language model, leveraging a novel linear attention architecture, on Amazon Web Services (AWS) Trainium2 accelerators. This advancement significantly overcomes limitations associated with traditional transformer models, particularly in managing long contexts during inference, leading to improved concurrency and reduced serving costs.
The Chenab Times has learned that i14’s Laplace 3B model utilizes a Gated Linear Attention (GLA) design coupled with the Muon optimizer, enabling its recurrent state to remain a consistent 29.5 MB per sequence, irrespective of the input’s context length. This efficiency was realized through the use of native PyTorch support via TorchNeuron Native, facilitating the model’s adaptation to AWS Trainium2, a specialized chip for machine learning workloads.
During training on a single trn2.48xlarge instance, Laplace 3B achieved an impressive throughput of approximately 120,000 tokens per second. For inference, the model demonstrated exceptional performance, serving 11,544 output tokens per second across 1,280 concurrent sequences. This technical collaboration highlights a significant step in optimizing large language model performance for demanding applications.
About i14 and Linear Attention
i14, an AI research and technology firm, focuses on creating model architectures, components, and runtimes inspired by artificial intelligence, computational cognitive science, and biological learning systems. Their objective is to develop models capable of retaining long-range context without the escalating serving state characteristic of conventional transformer architectures.
Traditional transformer models employ softmax attention, where computations scale quadratically with sequence length during training and the key-value (KV) cache grows with context during inference. This growth can impede concurrency, inflate serving costs, and compel agentic applications to truncate valuable historical data. Linear attention models, such as Laplace 3B, restructure computations around a recurrent state. This update process for each token maintains a fixed-size state, thereby achieving linear scaling for training attention (O(T)) and a constant memory footprint for inference state, regardless of context length.
The Laplace 3B model’s GLA design compresses historical information into a fixed-size matrix state. This approach contrasts with transformer softmax attention, which builds pairwise token scores with quadratic scaling during training and experiences expanding memory requirements with longer contexts during inference. While linear attention offers efficiency, it necessitates careful evaluation of retrieval quality for specific workloads. However, its fixed state is particularly advantageous for long-running agents, large-document analysis, and applications demanding persistent memory with predictable serving needs.
Adapting Laplace 3B for AWS Trainium2
The initial implementation of Laplace 3B was optimized for GPU using Triton kernels. The AWS Generative AI Innovation Center (GenAIIC) team collaborated with i14 to adapt the custom architecture for AWS Trainium2. This involved rewriting the Gated Linear Attention (GLA) kernel using standard PyTorch operators and leveraging the beta version of TorchNeuron Native, the native PyTorch backend for AWS Trainium, which offered superior performance.
Key adaptations included:
- Native PyTorch for GLA Kernel: The team replaced GPU-specific Triton kernels with standard PyTorch operators, segmenting the attention process into 16 chunks of 128 tokens each to align with the NeuronCore’s on-chip buffer partitions.
- Redesigned Muon Communication: Modifications to how the Muon optimizer communicates led to a significant reduction in per-rank traffic by 24 times and a 1.63-fold increase in throughput through two all_to_all collectives.
- Compiled Sequential Backward Pass: The sequential backward pass of the GLA recurrence was statically unrolled for all 16 chunks to accommodate the graph-size and on-chip memory limitations of the AWS Trainium2 compiler.
These changes were driven by the unique characteristics of AWS Trainium2 accelerators, including their software-managed on-chip memory, high-speed chip-to-chip interconnects, and a static compilation framework. Each NeuronCore on Trainium2 features a 28 MiB State Buffer (SBUF) with 128 partitions and a 2 MiB partial-sum buffer (PSUM), supporting efficient data processing.
Laplace 3B Architecture Details
Laplace 3B is a 24-layer decoder model where each layer substitutes softmax attention with Simple GLA. Each token’s vector undergoes normalization and a depth-wise causal Conv1d, followed by a QKVG projection that splits into queries, keys, values, and a gate. The GLA mixer then processes these through a compressed, decaying state representing past key-value pairs. The model incorporates a fixed decay rate per head, distributed geometrically to provide a spread of memory lengths across its 16 heads.
The feed-forward network (FFN) section follows the attention block, consisting of a GLU (SiLU) FFN with an inner dimension of 8,192. The model’s fixed state size ensures that inference remains O(1) per token, while training scales linearly with sequence length (O(T)), a marked improvement over the O(T^2) complexity of traditional transformers.
Key architectural parameters include 3.028 billion total parameters, 24 decoder layers, 16 heads with specific key and value dimensions, a GLA mixer with a fixed geometric decay schedule, a depth-wise causal Conv1d, and a GLU FFN. The model was trained on 100 billion tokens from FineWeb-Edu, using the Muon optimizer for 2D weight matrices and AdamW for embeddings and norms.
Engineering Challenges and Solutions
Adapting the model for Trainium2 presented specific engineering challenges. For GLA, re-expressing the kernel with standard PyTorch operators and aligning chunk sizes with SBUF partitions improved throughput. The Muon optimizer’s distributed nature, particularly under Fully Sharded Data Parallel (FSDP2), required optimizing communication. A naive gather-and-broadcast approach was replaced with an all_to_all transposition strategy in MuonFSDP2, drastically reducing per-rank traffic and increasing training throughput.
Furthermore, the backward pass for the sequential GLA recurrence required the AWS Trainium2 compiler to unroll the entire 16-step graph in advance. Careful selection of chunk count and size ensured the unrolled graph fit within scheduler limits and intermediates remained on-chip, avoiding spills to High Bandwidth Memory (HBM).
Training and Inference Performance
On a single trn2.48xlarge instance, Laplace 3B training consistently achieved approximately 120,000 tokens per second after compilation warmup. Loss values dropped smoothly from around 11.4 to 2.3729 by step 84,210 (approximately 44 billion tokens), with stable gradient norms indicating successful training dynamics.
For inference, the model, integrated with an unmodified vLLM engine, delivered 11,544 output tokens per second across 1,280 concurrent sequences. The constant serving state of 29.5 MB per sequence proved to be about 82 times smaller than a comparable KV cache at 8,192 tokens, with prompt length increases having a minimal impact on cost per output token. Benchmarking highlighted that operator dispatch and the cost of compiling new shapes under staggered arrivals were significant factors that fixed-batch benchmarks often overlook.
The performance gains were attributed to the efficient use of AWS Trainium2’s capabilities, including its native PyTorch support and specialized accelerators, enabling i14 to achieve state-of-the-art performance for its innovative linear attention model.
The Chenab Times News Desk