Most AI infrastructure procurement we are brought into starts with a chip and works backwards. The customer has decided on H100s or B200s, has a software stack that demands a certain interconnect, and only then asks where to put it. By the time we are in the conversation, the facility is the binding constraint and the workload has to bend.
We argue for the inverse: pick the facility that can credibly deliver the power, cooling, and network you need, then fit the chip choice into what is sourceable. This is unromantic. It is also how AI deployments stay on schedule.
Power: the actual numbers
A typical AI training rack today runs between 30 kW and 80 kW. The headline numbers from vendor data sheets are easy; the rack-level reality is harder.
A reasonable working baseline for an 8-GPU H100 SXM5 node:
- GPU thermal design power: 700 W per H100 SXM5 — 5.6 kW for 8 GPUs at peak.
- Host CPUs: ~500 W combined for a dual-socket node.
- Memory and NVMe: ~400 W combined.
- NIC and switch fabric: ~200 W per node, more on InfiniBand-heavy fabrics.
- Cooling overhead: 15–25% of IT load depending on facility PUE and rack-level design.
Per node total: 8–12 kW. Per rack with 4–8 nodes: 30–80+ kW. New-generation accelerators (B200, GB200 NVL72) push the per-rack numbers further — 120 kW+ in the densest configurations.
Most European colocation facilities are still designed around 5–15 kW per rack. The pool of facilities that can deliver 30+ kW per rack at the geographies our buyers want is significantly smaller. The pool that can deliver 60+ kW with the cooling story to back it up is smaller still. We track which is which.
Cooling: air, direct-to-chip, immersion
Air cooling stops being credible somewhere between 25 and 35 kW per rack, depending on hot-aisle containment and CRAH design. Above that, you need liquid.
Two viable production technologies:
Direct-to-chip (D2C) cooling. Cold plates on the heat-emitting components, coolant loop to a rear-door or row-level CDU. Deployable in retrofit conditions with the right facility cooperation. Most pragmatic path to 60+ kW today. We see this in newer hyperscale-adjacent halls and in selected Tier III+ enterprise facilities.
Immersion cooling. Whole-server submersion in dielectric fluid. Single-phase or two-phase. Higher density ceiling, lower mechanical complexity per kW, but the facility integration story is more involved (raised-floor incompatible, weight, fluid handling). Production-ready for greenfield AI builds; harder to retrofit.
Buyer-side question: which one can the facility you are evaluating actually deliver, today, in the rack count and growth profile you need? Public roadmaps are not procurement evidence.
Network: the part that decides whether your training run finishes
For multi-node AI training, the cluster network is in the critical path. Three layers matter:
- East-west fabric between GPUs. InfiniBand NDR (400 Gb/s) is the current production default for scale; 800 Gb/s is shipping. RoCE on Ethernet is viable for a subset of workloads but requires careful tuning. Latency and lossless behaviour matter more than headline bandwidth — get these wrong and your effective FLOPS drop by half.
- Storage fabric between compute nodes and parallel file systems. Often separated from the GPU fabric. NVMe-over-Fabrics or parallel filesystems on dedicated 200/400 Gb/s links is typical.
- External and inference fabric to peering and cloud on-ramps. AMS-IX, DE-CIX, AWS Direct Connect / Azure ExpressRoute / GCP Interconnect. Critical for inference deployment, less critical during training itself.
The network requirement turns into a facility filter as much as the power requirement does. Not every facility offers single-mode fiber routes that support 400 Gb/s in the cabinet count you need. We confirm this before introducing a buyer.
Storage: usually under-spec’d
Storage is the part of an AI build that gets cut to fit budget and then becomes the bottleneck three months in. A reasonable rule of thumb for a serious training cluster:
- Hot tier on NVMe local or NVMe-over-Fabric, sized for the working set. A 70B-parameter model checkpoint round-trip pattern wants this measured in tens of GB/s sustained, not in IOPS.
- Warm tier on parallel filesystem with intelligent caching. WekaIO, VAST, DAOS, GPFS — choose based on operational fit, not benchmark posters.
- Cold tier for datasets and snapshots. Object storage at lower cost per TB.
Cache-coherency and metadata-server scaling are the two failure modes we see most. Both require the facility’s network design to support the storage architecture, which loops back to the network point above.
How we filter
When a buyer comes in with an AI brief — through the RFQ form or directly — we filter facilities in this order: power density first, cooling capability second, network architecture third, location and price fourth. Facilities that cannot meet the first three drop out before we look at price. The shortlist is usually shorter than the buyer expects. That is the actual signal.