AWS Says DeepEP Over EFA Lifted MoE Rollout Throughput 40% on EKS

AWS has published a reference architecture for running large-scale reinforcement learning on Mixture-of-Experts (MoE) models using Amazon EKS, Elastic Fabric Adapter (EFA) and DeepEP, the expert-parallel communication library from the DeepSeek project. The headline claim: across 48 P5en instances split between training and inference, enabling DeepEP over EFA raised aggregate RL rollout throughput by 40 percent. AWS also says it has contributed upstream changes that move DeepEP's communication primitives onto libfabric, giving DeepEP v2 native EFA support.

The problem AWS is describing

The post frames post-training of sparse MoE models as an infrastructure balancing act rather than a compute problem. Reinforcement learning from human feedback and Group Relative Policy Optimization both require two very different workloads to run at the same time: rollout generation, which is large-scale distributed inference optimized for aggregate throughput rather than time to first token, and policy training, which behaves like pre-training, with tightly coupled workers advancing in lockstep. According to AWS, a straggling worker or a latency spike on the training side can stall the whole job or trigger NCCL timeouts, while insufficient rollout throughput leaves training accelerators idle.

MoE makes this harder in a specific way. AWS argues that as newer MoE architectures get sparser to cut inference cost, training becomes constrained by communication rather than compute. Expert Parallelism adds dynamic, fine-grained all-to-all token routing on top of the more structured traffic patterns of tensor, data and pipeline parallelism. Once a job spans more than one instance, that routing traffic shifts off the high-bandwidth intra-node NVLink and NVSwitch fabric and onto inter-node links, and the share of inter-node traffic grows as the expert parallelism degree grows.

That is a familiar observation in distributed training circles, but it is worth noting who is making it. A cloud provider has an obvious incentive to argue that the network, and specifically its own network adapter, is the bottleneck worth paying attention to.

What actually changed in DeepEP

DeepEP replaces generic all-to-all collectives with two specialized GPU kernels: a dispatch kernel that routes tokens from local GPUs to remote experts, and a combine kernel that gathers the processed tokens back. Intra-node transfers go over NVLink through NVSwitch; inter-node transfers need an RDMA-capable backend.

The substantive engineering contribution AWS describes is migrating those inter-node primitives from a CUDA-specific RDMA backend to libfabric. That makes the transport portable across libfabric-supported fabrics and, AWS says, reduces per-message overhead for the sparse, irregular traffic Expert Parallelism generates. On supported instance types including P5 and P6, EFA works with NVIDIA GPUDirect RDMA and OS bypass to move data directly between GPU memory buffers across instances. AWS also notes that NCCL 2.31 picks up recent EFA optimizations for dense collectives.

The portability angle matters more than it might look. A CUDA-specific RDMA path effectively assumes a particular interconnect stack; routing through libfabric is the kind of change that lets non-InfiniBand fabrics participate in an ecosystem built largely around them.

Reading the 40 percent figure carefully

The benchmark is described as 48 P5en instances, with 16 dedicated to training and 32 to inference, running what AWS calls a "super-sparse MoE model." Enabling DeepEP over EFA increased aggregate RL rollout throughput by 40 percent versus the same setup without it.

Several things are not disclosed. The model is not named, and no parameter count, expert count or expert parallelism degree is given. The absolute throughput numbers on either side of the comparison are not provided in the text, only the relative delta and a chart reference. The baseline configuration is not spelled out beyond the implication that it used standard all-to-all collectives. There is no reported effect on end-to-end time to convergence, on training step time, on tail latency or on run-to-run variance, and no cost comparison. The measurement window, number of runs and batch or sequence configuration are not stated.

That is a meaningful gap, because rollout throughput is an intermediate metric. In an asynchronous RL loop where training and inference must stay rate-matched, a 40 percent gain on the inference side translates into faster wall-clock convergence only if the training side can consume the extra experience. AWS's own framing of the balancing problem implies as much. Teams evaluating this should assume the number describes one subsystem, not a 40 percent cheaper RL run.

The cluster design is the reusable part

Independent of the benchmark, the topology AWS lays out is a reasonable template. The EKS cluster is split into separate node groups: GPU instances for rollout generation, reward model inference and policy training; CPU instances for environment simulation and preprocessing; and memory-optimized instances hosting experience buffers and checkpoint caches so that producers and consumers can exchange data without putting durable storage on the critical path. Checkpoints and finished artifacts land in Amazon S3.

The separation of concerns is the point. EKS handles scheduling, scaling and failure recovery across heterogeneous workers; NVLink and NVSwitch carry intra-node traffic; EFA carries latency-sensitive inter-node traffic. One operational constraint is easy to miss: AWS states that communicating nodes must sit in the same Availability Zone, and the EFA Kubernetes device plugin must be installed as a DaemonSet on the accelerator node group. Single-AZ placement has implications for capacity availability that the post does not address.

Spot for rollouts, on-demand for training

AWS recommends EC2 Spot Instances for rollout generation on the grounds that distributed inference partitions cleanly across independent workers. If a rollout worker is interrupted, its unfinished tasks return to the queue and get reassigned while other workers keep producing experience. Policy training, by contrast, stays on stable capacity because its workers must progress together.

The guidance comes with implementation requirements that teams often discover the hard way: rollout workers should process bounded units of work, publish completed samples frequently, and drain active requests when a Spot interruption notice arrives. AWS does not quantify the savings, and the design assumes a queue-and-reassign layer that the architecture diagram implies but the post does not detail. There is also an unaddressed question about policy freshness — if a Spot worker is reclaimed mid-rollout and its work is reassigned after a checkpoint update, the resulting samples may be generated under a stale policy.

Version pinning is the real operational tax

The reference stack is tightly pinned: EKS 1.31 or later, EFA installer 1.49 with the AWS OFI NCCL plugin, DeepEP 2.0.0, NCCL 2.31.2, SGLang 0.5.17, PyTorch 2.12.1 on CUDA 13.0, plus Miles 0.1.0, with DeepEP built from a specific commit. Supported instances listed include p5.48xlarge, p5e.48xlarge and p6-b200.48xlarge, with p4d, p4de and p6-b300 also named as usable for training or rollout.

AWS says keeping these components aligned is critical because GPU kernels, collective libraries and the EFA transport all contribute to end-to-end performance. Read another way: the 40 percent result is tied to a narrow matrix, and any team that upgrades one layer independently is off the tested path. Reproducing the numbers likely means reproducing the container, not just the architecture.

What to watch

• Whether the libfabric contributions land and stay maintained in upstream DeepEP, which determines if EFA support survives future releases without AWS-specific patches.
• Whether AWS or third parties publish end-to-end RL metrics — time to convergence, cost per training run, training-step latency — rather than a single rollout throughput delta.
• Disclosure of the benchmark model's architecture and expert parallelism degree, since the gain should scale with how much traffic crosses node boundaries.
• How much of the 40 percent holds on other instance families and at cluster sizes well above and below 48 nodes.
• Whether similar libfabric-based expert-parallel paths appear in other frameworks, which would suggest the optimization generalizes beyond this specific stack.