Modal, 8x H100 (serverless)

We ran our full test suite three times over three weeks on Modal's serverless H100s, and each run requested a full eight-GPU node (Modal's gpu="H100:8", meaning all eight cards on one host rather than a single GPU). Modal's serverless offering differs from a rented node in that, instead of handing you a machine to SSH into, it takes a container and a command, runs your code on GPUs it picks for you, and tears the container down once the job finishes. Since the GPU assignment varies from one run to the next, our runs landed on different physical nodes, and while two of the three came back nearly identical, the first differed sharply on a single measurement that we could not reproduce and now treat with caution.

Provisioning experience

Modal's serverless experience is quite different from renting a node yourself. You write a short function that asks for gpu="H100:8", run modal run, and the platform places eight H100s on one host, runs your code, and tears the container down when finished. The first run builds a container image (CUDA, nccl-tests, our suite, PyTorch), which takes a few minutes, after which the image is cached and the GPUs come up in seconds. On the third run we timed this directly with the image already built, and a cold call to a one-GPU function took 7.7 seconds round trip while an immediate second, warm, call took only 0.19 seconds. One nice benefit is that you don't have to remember to turn the instance off.

Getting it running did take a few rounds, since the default PyTorch install pulled a build for a newer CUDA than the host driver, and because the container is torn down at the end, our first results vanished with it, so anything you want to keep has to be written to a Volume, unlike a rented node whose disk persists as long as you hold it. Once that was sorted we had a pleasant experience, submitting the job, waiting, collecting the results, and paying only for the seconds the GPUs were busy.

What we found

On all three runs the silicon itself looked healthy and consistent, with matmul compute measuring 719 to 771 TFLOPS, the 300 second sustained burn completing cleanly every time with zero throttle samples, GPU memory bandwidth measuring about 2,790 GB/s, and NVLink peer-to-peer bandwidth sitting at 358 to 364 GB/s on every run. That NVLink bandwidth, measured across all GPU pairs, also confirms the eight cards were a single NVLink-connected node rather than eight scattered GPUs, and compute was never in question on any of the three.

Several findings held on every run, starting with the short training probe, which showed a severe straggler gap between the slowest and fastest GPU all three times, and even on the healthiest node the slowest step still ran to 82.7 ms against a fastest of 21.6 ms. The bulk NCCL collectives (all-gather, reduce-scatter, broadcast) came in between 235 and 259 GB/s and all-reduce around 290 GB/s, all below the 358 to 364 GB/s we measured on the raw NVLink peer-to-peer path, so the collectives were leaving bandwidth on the table, and the NVLink error counter read exactly 3,672 every time.

The three runs sit side by side below.

Runs 2 and 3 are near-twins, returning PCIe bandwidth identical to the decimal, the same link generation, and near-identical straggler and cooldown profiles, so either we landed on the same host twice or Modal's pool is fairly uniform. Both presented Gen4 x16 links rather than the Gen5 we expect of an H100 SXM host, and that reading comes from the maximum PCIe link generation nvidia-smi reports rather than the current link, which idles down to Gen1 to save power, so it reflects the ceiling the driver reports rather than a momentary downclock. Because this is a serverless container, though, it is the ceiling the virtualization layer exposed to the guest rather than necessarily the physical slot, and if Modal caps or misreports the link we would read Gen4 even on a Gen5 board. At about 26 GB/s these two nodes were roughly saturating what a Gen4 x16 link can carry, so the honest reading is a slower presented link generation rather than an actively throttled transfer, and the fact that run 1 instead presented Gen5 means the three runs did not even share a link generation, which is further sign they were different physical nodes.

Run 1 is the exception, and its device-to-host reading of 1.7 GB/s is the single most extreme number in the report, yet it never happened again, since a 1x validation run on the same platform the same day measured a normal 26.3 GB/s and every later run measured about 26 GB/s. We hold back from trusting the 1.7 for two reasons, the first being that it never reproduced, which on a serverless platform where each run is a fresh container makes one reading that four others contradict more likely a transient or probe artifact than a stable property of the hardware, and the second being that nothing independent confirms it, since the bandwidth tool we added as a cross-check never produced data on any run and the training probe's own transfer figure ran low across all three runs and so cannot single run 1 out. The node was somewhat slow in absolute terms, its training probe running about twice as slow as the other two at 10,698 tokens per second, so it was not perfectly healthy, but we would not stand behind the specific 1.7 figure or want a reader to anchor on it.

Thermals split by node, with run 1's being pristine while runs 2 and 3 both showed degraded cooldown behavior, their exponential fit poor (r² of 0.74 and 0.76, against 0.97 on run 1) and the ambient reading at the end of cooldown sitting at 39 to 40°C. Their tau was lower, which on its own would suggest faster cooling, but the poor fit quality and the elevated ambient point to less stable airflow around those nodes rather than to better cooling.

On Modal's serverless GPUs you cannot choose the node you land on, and in our small sample the three nodes were more alike than not. The findings that reproduced across runs were a host-transfer path a generation slower than the platform class implies, a training straggler on every run, collective bandwidth well under the raw NVLink link, and cooldown instability on two of the three nodes, while one run additionally posted an extreme device-to-host reading that we could not reproduce or corroborate. Of all of these the straggler was the most consistent, turning up on every run.

Running it yourself

The method is described in How we benchmark providers, and the harness is on PyPI as sixtytwo-cli. All three runs used sixtytwo test --full --json inside a Modal container built from nvidia/cuda:12.6.2-devel-ubuntu22.04, with nccl-tests, a driver-matched PyTorch cu126 build, and sixtytwo-cli[gpu]. A small modal run wrapper builds the image, runs the suite, and saves the report to a Volume.