AWS and Qumulo Put Training Compute in One Region and the Data in Another

GPU capacity and training data rarely land in the same AWS Region, and teams have generally had to choose between replicating petabytes or eating wide-area latency on every read. In a post on the AWS Machine Learning blog, AWS and Qumulo describe a third option: keep one copy of the dataset in a hub Region and let a SageMaker HyperPod cluster in another Region read it through a predictive caching layer. Their validation run shows a remote cluster matching a co-located one on throughput — with an asterisk on scale that anyone evaluating this should read closely.

The setup

According to the post, the architecture pairs Amazon SageMaker HyperPod with Cloud Native Qumulo (CNQ) and Qumulo's Cloud Data Fabric (CDF). A single authoritative copy of the dataset lives on a CNQ "hub" instance in one Region. A CNQ "spoke" instance in a second Region presents the same POSIX namespace, and HyperPod compute nodes mount that local spoke over NFS. The spoke pulls data from the hub across a VPC peering connection, so compute nodes never issue cross-Region reads themselves.

AWS says CDF replicates filesystem metadata to the spoke first, which makes the namespace browsable within seconds before any file data moves. Reads are then served from the nearest valid source: local NVMe on a cache hit, the hub on a miss. The component doing the anticipating is NeuralCache, described as a model that predicts which 4 KB blocks a data loader will request next and prefetches them into the spoke's local NVMe. The post also notes that CDF uses congestion control paced to measured bottleneck bandwidth and round-trip time rather than backing off on packet loss, which it credits for sustaining near-line-rate throughput on paths with round-trip times as high as 900 ms.

Practically, the appeal is that nothing in the training code changes. The same mount path works in both Regions, the same lifecycle script mounts storage at node startup, and the same Kubeflow PyTorchJob manifest runs in either place.

What was measured

The validation ran the same job independently on two clusters: a hub in US East (Ohio) co-located with the data, and a spoke in US West (Oregon) reading remotely with 60 ms of network latency between them. The workload was a 1.02-billion-parameter LLaMA v3 model on two ml.p5.48xlarge instances per cluster — 16 H100 GPUs in total — using PyTorch 2.1 with Fully Sharded Data Parallel, batch size 24 per GPU and 4,096-token sequences. The dataset was C4, tokenized for LLaMA into 48.4 million sequences packed to 4,096-token windows.

The reported results: the hub sustained 116–117 samples per second and hit 999 batches in 18.5 minutes. The spoke with a warm cache ran 115–116 samples per second and also reached 999 batches in 18.5 minutes. A cold-start spoke — the worst case, with no prior runs — ran 95–115 samples per second before converging to 116, taking roughly 19–20 minutes. AWS says batches 0–150 run 15 to 20 percent slower during NeuralCache warmup, then converge within the first epoch.

On the storage side, the post reports the hub delivering 1.0–1.3 GBps sustained reads at 2–3 ms read latency with 98–100 percent GPU utilization, and says CNQ saturated a 10 Gbps network link with a four-node cluster of r5.8xlarge instances. Once the spoke cache is warm, AWS says 94–96 percent of reads are served locally at sub-5 ms latency.

AWS also publishes an extrapolation table for the one-time cold-start penalty: 0.81 percent of wall-clock time at 10,000 batches, 0.08 percent at 100,000, and 0.008 percent at one million. The post explicitly labels the 100,000- and one-million-batch rows as extrapolated from the observed cold-start duration, and says actual results vary by dataset size, access pattern and network conditions.

What the benchmark does not establish

The honest framing here is that this is a vendor-authored architecture post with a small, tightly controlled validation. Sixteen H100s and a roughly one-billion-parameter model is a plausible integration test, not a frontier training run. Nothing in the post demonstrates that the same behavior holds at hundreds or thousands of GPUs, where aggregate read bandwidth demand scales linearly and a single spoke's cache and uplink become the contended resource. The post does not disclose how the approach behaves at larger cluster sizes.

There is also a detail worth reconciling. The post claims "p5.48xlarge network saturation" as an outcome while describing storage throughput of 1.0–1.3 GBps saturating a 10 Gbps link. Those are very different bandwidth regimes, and the post does not explain the relationship between them. For teams sizing this, the relevant number is the one that matters for their data loader, and it is not spelled out.

A few other gaps stand out. The dataset's size on disk is not disclosed, which matters because a 94–96 percent hit rate strongly implies the working set fits in the spoke's local NVMe. If it does not, the warm-cache numbers may not transfer. Costs are absent entirely: no pricing for CNQ, no figures for cross-Region VPC peering data transfer, and no comparison against the replication approach this is meant to replace. The post does not report run-to-run variance or repeated trials — the results are presented as single comparisons.

Write behavior is another thin spot. The post notes that periodic IOPS spikes on the hub dashboard correspond to checkpoint writes, and the job manifest points checkpoints at the same Qumulo mount, but it does not explain how spoke-side writes reconcile back to the hub, what the consistency semantics are for a shared ReadWriteMany namespace across Regions, or what happens to a training job if the peering link degrades mid-run. HyperPod's own resilience features — automatic health checks, node replacement and checkpoint recovery — are cited, but those address compute failures, not storage-path failures.

Finally, predictive caching works best on predictable access. AWS says NeuralCache identifies the sequential pattern FSDP training produces. Pipelines with aggressive global shuffling, curriculum sampling or heavy on-the-fly augmentation may be harder to predict, and the post does not test those cases.

The operational picture

The walkthrough is conventional AWS plumbing: deploy CNQ from AWS Marketplace via CloudFormation in the hub Region, create and accept a VPC peering connection, add reciprocal routes, and restrict security groups to NFS on TCP 2049. AWS notes that cross-Region VPC peering traffic is encrypted in transit. HyperPod clusters are created in both Regions with EKS as the orchestrator, each in the same VPC as its EKS control plane and on private subnets. Storage is mounted through the Qumulo CSI driver installed via Helm, backed by a static PersistentVolume and claim.

One tuning choice is worth flagging: the job runs 64 data loader workers with a prefetch factor of 4, keeping 256 batches in flight. AWS says this masks latency during cache warmup while FSDP overlaps gradient synchronization with I/O. That is a meaningful amount of prefetch buffering, and it is part of why the cold-start penalty stays as small as it does.

AWS also suggests skipping the cold start altogether by issuing a pre-warm request through the Qumulo API from an orchestration script before training begins, and points to multi-spoke deployments, hybrid setups using Direct Connect for on-premises data, and archiving checkpoints to S3 Glacier as extensions. None of those variants were benchmarked in the post.

What to watch

• Whether AWS or Qumulo publish results at cluster sizes beyond two nodes, where a single spoke's cache and cross-Region uplink face real contention.
• Total cost of ownership versus replication: CNQ licensing, spoke NVMe capacity and cross-Region transfer charges were not disclosed, and that comparison is the actual decision.
• Write-path and consistency details for checkpoints and any spoke-originated writes, which the post leaves unspecified.
• How NeuralCache holds up under shuffled or non-sequential data loaders, versus the sequential FSDP pattern it was validated on.
• Whether independent benchmarks reproduce the warm-cache parity claim, since the only figures available come from the vendors involved.