The map
Four layers, from the building to the scheduler
Almost every AI infrastructure bottleneck lives in one specific layer. Pinpointing which one is half the diagnostic work — and it stops you adopting software that solves a problem you do not have.
The label on the right is the service we provide at that layer. Inside each section below we also state how far our own evaluation work genuinely goes — we would rather tell you where our depth ends than have you discover it halfway through an engagement.
Orchestration, scheduling and serving
The software deciding which job runs on which accelerator, how requests are served, and what happens when a node disappears.
The question engineering teams ask
How many requests can one accelerator actually serve, how do we keep shared capacity busy across teams, and what happens to a running job when a spot node is reclaimed?
What goes wrong here
Inference cost climbs faster than usage because serving was left at defaults. Reserved capacity sits idle while another team queues for it. A reclaimed spot instance loses a day of training because the checkpoint was written to the node that disappeared.
How far we go — Deep evaluation
We evaluate products at this layer ourselves and advise on them directly.
This is where we started and where our evaluations are most thorough.
Products we have written up at this layer
vLLM
Inference serving engine
An open-source engine for serving large language models, with memory handling built for high concurrent throughput.
- Fit
- Teams self-hosting an open-weight model for production inference.
- Catch
- Serves models well; does not manage your fleet, routing or tenancy.
LiteLLM
Inference gateway
A gateway presenting one consistent interface across many hosted model providers and your own self-hosted models.
- Fit
- Applications calling more than one model provider, or planning to.
- Catch
- Adds a network hop, and sits on the critical path of every request.
SkyPilot
Multi-cloud workload orchestration
An open-source layer that runs the same job on whichever cloud, specialist GPU provider or Kubernetes cluster has capacity.
- Fit
- Teams using more than one compute provider, or wanting to.
- Catch
- You still need accounts and quota with every provider it places work on.
Run:ai
GPU pooling and fractional sharing
A Kubernetes-based orchestration layer that pools accelerators across teams and allocates fractions of one to a workload.
- Fit
- Shared clusters where several teams compete for fixed capacity.
- Catch
- Assumes you already run Kubernetes, and run it well.
Kubernetes batch scheduling
Batch scheduling
Add-ons giving Kubernetes the queueing and gang-scheduling behaviour AI workloads need and plain Kubernetes does not provide.
- Fit
- Organisations already standardised on Kubernetes.
- Catch
- You assemble a scheduler from components rather than buying one.
Compute and cloud
Where the accelerators physically are — a hyperscaler, a specialist AI cloud, or your own racks.
The question engineering teams ask
Where does our capacity come from, on what term, and does the total cost of a multi-year reservation actually beat staying flexible?
What goes wrong here
A multi-year term signed for a workload that looked different within two quarters, with no way to hand it back. Or on-demand rates paid indefinitely for a baseline that has been flat for a year.
How far we go — Deep evaluation
We evaluate products at this layer ourselves and advise on them directly.
We track the specialist providers closely, including how their commercial terms differ from the hyperscalers.
Products we have written up at this layer
AWS, Google Cloud and Azure
Hyperscale cloud
The three large general-purpose clouds, each offering accelerated compute alongside everything else you already run.
- Fit
- Estates where the data, identity and compliance posture already live.
- Catch
- Generally the highest hourly rate, and quota is often the real limit.
CoreWeave
Specialist AI cloud
A cloud built specifically for accelerated workloads at scale, rather than a general-purpose cloud with GPUs added.
- Fit
- Sustained large-scale training where committed capacity makes sense.
- Catch
- Oriented to larger commitments than intermittent workloads justify.
Lambda
Specialist AI cloud
A GPU cloud aimed at AI engineering teams, which also sells hardware for on-premises deployment.
- Fit
- Teams wanting specialist pricing, and those weighing rent against buy.
- Catch
- Popular hardware types can be capacity-constrained at times.
RunPod
On-demand and serverless compute
An on-demand accelerator platform with per-second billing and a serverless mode for inference.
- Fit
- Development, experiments, and inference that scales to zero.
- Catch
- Confirm the guarantees carefully before running anything under a strict SLA.
Network and interconnect
The private links between your data centre, your clouds, your storage and everyone else.
The question engineering teams ask
What is cross-cloud egress actually costing us each month, and is the path between storage and accelerators wide enough to keep them fed?
What goes wrong here
Egress charges leak quietly until data movement is one of the largest lines on the invoice. Accelerators idle waiting on input because the path to the data was never sized for this workload.
How far we go — Architecture and design
We specify the requirement and review the design. Delivery is with a specialist partner.
We size and specify the requirement; delivery is with a specialist partner.
Products we have written up at this layer
Megaport
Network as a service
A software-defined network service for creating private connections between data centres and cloud providers on demand.
- Fit
- Hybrid estates moving enough data for transfer charges to matter.
- Catch
- Only useful where they are already present in your facility.
Equinix Fabric
Interconnection
On-demand interconnection between Equinix facilities, cloud providers and other participants in them.
- Fit
- Organisations with equipment already in an Equinix facility.
- Catch
- Compelling if you are already there; a weak reason to move.
Facility, power and cooling
The building, the power feed and the cooling a dense accelerator rack needs to run at the performance you paid for.
The question engineering teams ask
Can our facility or colocation footprint take the power draw and heat of the racks we are planning, and is the software layer above them ready for it?
What goes wrong here
Hardware arrives and the room cannot cool it. Accelerators throttle under sustained load, so you pay for performance that is never delivered — and nobody finds out until the first long run.
How far we go — Requirements and constraints
We map the constraints this layer imposes on everything above it. We have not run product evaluations here yet.
We are honest about this. We map the constraints and coordinate specialists; we are not facility engineers and we do not pretend to be.
Products we have written up at this layer
Schneider Electric
Power distribution and facility management
Power distribution, UPS, cooling and data centre infrastructure management software.
- Fit
- Facilities being assessed or upgraded for high-density racks.
- Catch
- Facility decisions are long-lived and expensive to reverse.
LiquidStack
High-density liquid cooling
Specialist liquid cooling systems, including immersion and direct-to-chip, for rack densities air cannot handle.
- Fit
- Facilities being retrofitted or built for modern accelerator density.
- Catch
- Changes how the facility is operated, serviced and warranted.
Tell us what you are trying to solve.
Describe your current setup and the problem in your own words. We reply within 2 working days with an honest assessment of whether we can help.