8x AMD Radeon AI PRO R9700, One Fabric

What We Just Lit Up

The AMD Radeon AI PRO R9700 is one of the most interesting cards to land on our bench this year. It is a 32GB GDDR6, PCIe 5.0 x16, 300W RDNA 4 workstation GPU that AMD officially specs for up to 4-way multi-GPU deployments. Multiple partner and press sources repeat that "4-way" number as the practical ceiling.

We just ran eight of them as a single fabric inside a PRU 2500 — and every GPU-to-GPU pair is moving data at line-rate PCIe 5.0 x16, bidirectional. That is not a small footnote. It changes what an R9700 fleet can do.

BY THE NUMBERS

111 GB/s

Bidirectional, every GPU-to-GPU pair — no exceptions

97 GB/s

CPU↔GPU bidirectional, per link, symmetric on both sockets

56.8 GB/s

GPU→GPU per-direction average — near PCIe 5.0 x16 theoretical

93–96%

Fabric-port link utilization, ingress and egress, under full-mesh PyTorch

<1%

Host uplink utilization — the switches do the work, not the CPUs

2×

Effective GPU-to-GPU bandwidth vs. NVIDIA's own RTX PRO 6000 reference architecture

The Measurement

Below is the peak bidirectional bandwidth captured across 2 CPUs and 8 R9700 GPUs. The GPU↔GPU block is what matters for multi-GPU training and inference — read/write concurrent. GPU-quad separators in the source heatmap show no cross-domain cliff.

Three things jump out of that heatmap:

Every off-diagonal GPU-to-GPU pair sits at ~111 GB/s bidirectional. Not just neighbors on the same PCIe switch. Not just cards on the same NUMA domain. Every pair. The fabric is flat.
Same-quad vs. cross-quad is a rounding error. GPU 00–03 talking to GPU 04–07 lands at 111.16–111.19 GB/s — indistinguishable from same-quad traffic at 111.94–111.97 GB/s. There is no cross-domain cliff.
CPU↔GPU bidirectional is ~97 GB/s per link across all eight GPUs, and both CPUs see it symmetrically. Host-side staging is not the bottleneck either.

The corresponding GPU→GPU per-direction average is 56.80 GB/s — the per-direction half of that bidirectional total, exactly what you would expect from a PCIe 5.0 x16 link running near theoretical (~63 GB/s per direction). We are effectively pinning the PCIe Gen 5 physical layer on every GPU, in every direction, simultaneously.

Under Real Load: All 8 GPUs, All-to-All, at Once — Over PyTorch

Pairwise peak numbers are one thing. What happens when every GPU is talking to every other GPU at the same time, over PyTorch, is the number that decides whether you actually have a training-class fabric.

So we ran that too — a full-mesh all-to-all across all eight R9700s under PyTorch — and grabbed live per-port statistics from the Broadcom PCIe 5 switch on each of the two motherboards. The port map is straightforward: fabric ports 16/32/48/128 fan out to one quad of GPUs, ports 64/80/96/112 fan out to the other, port 0 is the host uplink.

What the data actually shows — and one of these findings is genuinely rare:

Every fabric-facing port on both switches is running at 93–96% link utilization, ingress and egress, simultaneously. That is what "full mesh" is supposed to mean and almost never actually looks like on a bench. Not one port dips, not one link is idle.
Per-port payload rates land at ~50–51 GB/s in every direction. The Broadcom counters split by ingress and egress, so the honest single-direction number is right there — ~55–56 GB/s of total data per port each way. The pairwise number and the full-mesh-under-PyTorch number agree.
The host uplink (port 0) is idle at <1% utilization on both switches. That is the definition of a GPU-to-GPU fabric doing the work instead of the host. The CPUs are not staging bytes for this traffic — the switches are.
The fabric is uniform. Across 16 fabric ports and 32 direction-counters, the tightest spread is 93.20–95.67% — a 2.5-point window. No outlier port, no cliff, no direction that lags.
This is running on PyTorch, on AMD ROCm, on 9700s. No CUDA, no NVLink, no proprietary interconnect. Open stack, open framework, commodity workstation silicon — behaving like a real training fabric.

The pairwise heatmap proves the ceiling. The live port capture proves the fabric holds the ceiling under all-to-all load. Those are two different claims, and both are now on the record.

How This Compares to the NVIDIA RTX PRO 6000

The NVIDIA RTX PRO 6000 Blackwell Workstation Edition is a 96GB GDDR7, PCIe 5.0 x16 card. It has more per-card memory and more per-card memory bandwidth than the R9700 — full stop. It is also a card without NVLink. In a standard multi-GPU host, RTX PRO 6000 GPU-to-GPU traffic falls back to PCIe and — when it crosses sockets — to the inter-CPU link.

NVIDIA's own developer blog on ConnectX-8 SuperNICs quantifies exactly this: in a traditional dual-socket RTX PRO 6000 design, GPU-to-GPU across two CPU sockets is limited to ~25 GB/s or less by inter-CPU link utilization, and their fix is to route NCCL traffic out through the network to get back to ~50 GB/s per GPU.

Read that again. NVIDIA's recommended architecture for scaling RTX PRO 6000 across a host is to leave the host — pay the NIC tax, pay the network tax, pay the topology complexity — to get to ~50 GB/s per GPU of inter-GPU IO.

GPU-to-GPU Bandwidth: 8x R9700 in a PRU 2500 vs. Standard Dual-Socket RTX PRO 6000

Metric	8x R9700 (PRU 2500)	RTX PRO 6000 (dual-socket + SuperNIC fix)
GPU-to-GPU bandwidth, every pair	~111 GB/s bidirectional, inside the chassis	~50 GB/s per GPU, routed out through the network
Cross-socket GPU-to-GPU, no NIC	No cross-domain cliff — matches intra-quad	~25 GB/s or less, limited by inter-CPU link
Path to get there	Photonic PCIe fabric, native inside the chassis	SuperNIC + network switch, out of the host
Vendor multi-GPU spec ceiling	AMD specs 4-way; we ran 8-way at line-rate	No NVLink on this card

Our 8x R9700 fabric is doing ~111 GB/s bidirectional between every GPU pair inside the chassis — and holding it under full-mesh PyTorch traffic, with every switch port pinned in a tight 93–96% link-utilization band in both directions at once. No SuperNIC in the path, no cross-socket penalty, no per-flow topology surprise. That is roughly 2× the effective GPU-to-GPU bandwidth NVIDIA's own reference architecture targets for their comparable workstation-class Blackwell card — and it happens on a card AMD themselves only officially spec at 4-way.

It is also ahead of what we published for RTX 5090 fleets in the same PRU 2500 chassis, where GPU-to-GPU landed at roughly 102–105 GB/s bidirectional before tuning. Same chassis, different silicon, cleaner numbers.

What Made This Possible: The PRU 2500 as a Photonic Resource Unit, Not a Slot Count

The reason AMD's spec sheet says "4-way" is not a limitation of the R9700 silicon. It is a limitation of what a normal workstation motherboard can physically present: how many PCIe 5.0 x16 slots it has, at what electrical width, on which NUMA domain, cooled by what airflow.

The PRU 2500 is Corespan's Photonic Resource Unit — a composable, direct-liquid-cooled GPU utility chassis that decouples GPU capacity from the host motherboard. Paired with the FIC 2500 host fabric card and orchestrated by Corespan Composer, the PRU 2500 lets the host carry CPU, memory, and networking while the chassis carries the GPUs, connected over a photonic PCIe fabric that presents every GPU to every host as native PCIe 5.0 x16.

For the R9700 specifically, three PRU 2500 properties are doing the work in this test:

Line-rate PCIe 5.0 x16 to every GPU, symmetrically. No switch oversubscription, no lane sharing, no negotiated downshift when the eighth card lights up. That is why every off-diagonal pair sits at 111 GB/s instead of tapering as you fan out.
A flat GPU-to-GPU topology. There is no "same quad fast, cross quad slow" cliff, because the fabric is not a tree of tiny PCIe switches bolted onto a consumer chipset. Same-quad and cross-quad look identical in the heatmap.
300W slot power. The R9700 ships as a 300W dual-slot blower designed for dense workstation stacks. Eight of them at full duty in a chassis designed for 700W GPUs is not a problem.

Composer sits on top of all of that and treats the eight R9700s as a pool — assignable to one host as a single 8-GPU shape, or split across multiple hosts as smaller shapes, without re-cabling.

Why R9700 Buyers Should Care

The R9700 is compelling for exactly the reason AMD is pitching it: 32GB of GDDR6 for $1,299 MSRP, with an open ROCm stack and native PCIe 5.0. It is one of the best price-per-VRAM cards you can buy for local and small-cluster AI right now.

The question every R9700 buyer runs into next is scaling. AMD's own guidance stops at 4-way with 128GB pooled VRAM. Beyond that, in a conventional server, you are back to the same problem NVIDIA RTX PRO 6000 owners have — PCIe topology cliffs, cross-socket penalties, and increasingly heroic networking to make GPU-to-GPU traffic look like a fabric.

The PRU 2500 answer is: do not build the heroic networking. Put eight R9700s in a PRU 2500, keep every GPU-to-GPU pair at 111 GB/s bidirectional, keep every card at sustained boost, and let Composer allocate the pool.

For a 32GB-per-card design point, that turns the R9700 from a "great single-workstation AI card" into a 256GB pooled-VRAM utility cell with a fabric that wins.

If you are sizing an R9700 fleet — or you were already sizing an RTX PRO 6000 fleet and the GPU-to-GPU numbers above got your attention — we should talk.

In the Lab: 8x AMD Radeon AI PRO R9700, One Fabric