Solution
Our inference costs rise faster than our usage
The model is in production and working. The monthly bill is growing faster than the business it supports.
For most organisations running AI in production, continuous inference has overtaken training as the larger and less predictable line item. Training is a project with an end date. Serving is a running cost that scales with every user, every session, and — with agent-style workloads — every step an agent takes on their behalf.
The uncomfortable part is that inference cost is mostly an engineering variable, not a procurement one. Two teams serving the same model on the same hardware can differ several-fold in cost per request, and the difference is in batching, memory handling, routing and model selection rather than in the price they negotiated.
What is usually going on
Serving efficiency was never tuned
A model served with default settings handles a fraction of the concurrent requests it could. Since you pay for the GPU by the hour regardless, throughput per GPU is the number that decides the bill.
The largest model is used for every request
Most production traffic is routine. Routing simple requests to a smaller or cheaper model, and reserving the frontier model for the requests that need it, is usually the single largest saving available.
Agent workflows multiply calls invisibly
One user action can become a chain of model calls. Cost per request looks fine; cost per completed task is what the business actually pays, and few teams measure it.
Capacity provisioned for peak, paid for at all hours
Inference demand is spiky. Fixed capacity sized for the worst hour is idle for most of the day unless something is scaling it down.
Nobody owns the number
Inference spend often sits between engineering and finance, with neither holding a target. Cost per thousand requests, tracked weekly, changes behaviour on its own.
What we look at, in order
- 01 Establish your real unit economics — cost per request, per session, and per completed task
- 02 Review serving configuration and measure achievable throughput per GPU against current throughput
- 03 Assess whether request routing across model sizes and providers is viable for your traffic mix
- 04 Separate latency-critical traffic from work that can be batched or deferred
- 05 Model the change before committing to it, including the engineering effort to get there
What you receive
- Current unit economics baseline, with the measurement method documented
- Serving architecture recommendation with projected cost per request
- Routing and model-tiering design where it applies
- A costed comparison of options, including doing nothing
Delivered as a written assessment. More on how we work.
Products relevant to this
Not a shortlist for you specifically — that depends on constraints this page cannot know. These are the options an engineering team addressing this problem will encounter first.
vLLM
L4 · Inference serving engine
An open-source engine for serving large language models, with memory handling built for high concurrent throughput.
- Fit
- Teams self-hosting an open-weight model for production inference.
- Catch
- Serves models well; does not manage your fleet, routing or tenancy.
LiteLLM
L4 · Inference gateway
A gateway presenting one consistent interface across many hosted model providers and your own self-hosted models.
- Fit
- Applications calling more than one model provider, or planning to.
- Catch
- Adds a network hop, and sits on the critical path of every request.
RunPod
L3 · On-demand and serverless compute
An on-demand accelerator platform with per-second billing and a serverless mode for inference.
- Fit
- Development, experiments, and inference that scales to zero.
- Catch
- Confirm the guarantees carefully before running anything under a strict SLA.
AWS, Google Cloud and Azure
L3 · Hyperscale cloud
The three large general-purpose clouds, each offering accelerated compute alongside everything else you already run.
- Fit
- Estates where the data, identity and compliance posture already live.
- Catch
- Generally the highest hourly rate, and quota is often the real limit.
Other problems
Tell us what you are trying to solve.
Describe your current setup and the problem in your own words. We reply within 2 working days with an honest assessment of whether we can help.