Solution
We are weighing model APIs against running our own
Provider invoices have grown past the point where self-hosting on dedicated accelerators is worth costing properly.
There is a volume at which paying per token stops being the cheaper option and starts being the expensive one. Below it, a hosted API is almost always the right answer — no infrastructure to run, no capacity to hold, no on-call. Above it, the same workload served on dedicated accelerators using an open-weights model can cost a fraction of the invoice.
The difficult part is that the break-even is specific to your traffic. It moves with request volume, prompt and output length, how spiky demand is, whether requests can be batched, and how much engineering time you are prepared to spend keeping a serving stack healthy. Working it out properly is a modelling exercise, not a matter of opinion — and it is worth doing before the migration rather than during it.
What is usually going on
The comparison is made on headline token price alone
Self-hosting does not cost the accelerator rate — it costs the accelerator rate divided by how many requests you can actually serve per hour on it, plus the engineering time to keep it running. Both sides need the full number.
Idle hours are ignored in the model
A hosted API charges nothing when nobody is using it. A reserved accelerator charges the same at 3 AM as at peak. If demand is spiky, that gap can erase the saving entirely.
Quality is assumed rather than measured
Moving to an open-weights model is a quality decision as much as a cost one. Without an evaluation set that reflects your actual traffic, the comparison is not a comparison.
Serving throughput left at defaults
How many concurrent requests one accelerator handles varies enormously with the serving engine and its configuration. This single number decides whether self-hosting is cheaper at all.
The operational cost is left out
Somebody has to patch it, watch it, scale it and be paged for it. That is a real and recurring cost, and it belongs on the self-hosted side of the ledger.
What we look at, in order
- 01 Establish current spend and traffic shape — volume, token lengths, concurrency, how spiky demand actually is
- 02 Measure achievable throughput per accelerator for candidate models and serving configurations
- 03 Model both options end to end, including idle time, engineering effort and on-call
- 04 Identify which traffic should move and which is better left on a hosted API
- 05 Design the migration so it is reversible, and can run both paths side by side while you compare
What you receive
- Break-even analysis with your traffic profile and the assumptions stated
- Recommended serving architecture and expected throughput per accelerator
- A traffic-split recommendation rather than an all-or-nothing answer
- A reversible migration sequence, with the fallback path defined
Delivered as a written assessment. More on how we work.
Frequently the honest answer is a split — the high-volume, predictable traffic self-hosted and the rest left on a hosted API. If the numbers say stay where you are, that is what the assessment will say.
Products relevant to this
Not a shortlist for you specifically — that depends on constraints this page cannot know. These are the options an engineering team addressing this problem will encounter first.
vLLM
L4 · Inference serving engine
An open-source engine for serving large language models, with memory handling built for high concurrent throughput.
- Fit
- Teams self-hosting an open-weight model for production inference.
- Catch
- Serves models well; does not manage your fleet, routing or tenancy.
LiteLLM
L4 · Inference gateway
A gateway presenting one consistent interface across many hosted model providers and your own self-hosted models.
- Fit
- Applications calling more than one model provider, or planning to.
- Catch
- Adds a network hop, and sits on the critical path of every request.
RunPod
L3 · On-demand and serverless compute
An on-demand accelerator platform with per-second billing and a serverless mode for inference.
- Fit
- Development, experiments, and inference that scales to zero.
- Catch
- Confirm the guarantees carefully before running anything under a strict SLA.
Lambda
L3 · Specialist AI cloud
A GPU cloud aimed at AI engineering teams, which also sells hardware for on-premises deployment.
- Fit
- Teams wanting specialist pricing, and those weighing rent against buy.
- Catch
- Popular hardware types can be capacity-constrained at times.
Other problems
Tell us what you are trying to solve.
Describe your current setup and the problem in your own words. We reply within 2 working days with an honest assessment of whether we can help.