Solution
It works in the pilot but not in production
The prototype proved the idea. Turning it into something dependable has stalled.
This is the most common place AI initiatives stop. A prototype built on a laptop or a single rented GPU demonstrates that the idea works. Then the questions arrive that a prototype never had to answer: what happens under real concurrency, who is on call, how do we roll back a bad model version, where does the data actually live, and what does this cost per month at ten times the volume.
These are infrastructure and operations questions rather than model questions, and they are usually the reason the gap takes far longer to cross than anyone estimated. The work is unglamorous and entirely tractable — it just needs to be named and sequenced rather than discovered one incident at a time.
What is usually going on
No deployment path, only a notebook
Nothing is reproducible. There is no build, no versioned artefact and no way to get a change into production twice the same way.
Never tested at real concurrency
A demo serves one request at a time. Production serves many at once, and the failure mode under load is rarely what anyone predicted.
No rollback for the model itself
Application code has version control and rollback. The model version, the prompt and the retrieval index frequently do not, so a bad change cannot be reversed quickly.
Data access designed for the demo
A prototype reads a copied dataset. Production needs live data, with access control, residency and lineage that the prototype never considered.
Nobody has agreed what "working" means
Without an availability target and a latency budget agreed in advance, there is no way to tell whether the system is ready, and no way to size the infrastructure for it.
What we look at, in order
- 01 Write down the actual production requirements — availability, latency budget, expected concurrency, growth
- 02 Map the gap between the prototype and those requirements, item by item
- 03 Choose the smallest deployment and orchestration platform that meets them, not the most capable
- 04 Design the release path: versioning, staged rollout, rollback, and who is paged
- 05 Sequence the work so something dependable ships early rather than everything shipping late
What you receive
- A written production readiness assessment against agreed targets
- Target architecture across compute, orchestration and data access
- A prioritised gap list with effort and sequence
- Platform and tooling shortlist with trade-offs stated
Delivered as a written assessment. More on how we work.
If the blocker turns out to be model quality rather than infrastructure, we will say so. That is a different problem and we are not the right people for it.
Products relevant to this
Not a shortlist for you specifically — that depends on constraints this page cannot know. These are the options an engineering team addressing this problem will encounter first.
vLLM
L4 · Inference serving engine
An open-source engine for serving large language models, with memory handling built for high concurrent throughput.
- Fit
- Teams self-hosting an open-weight model for production inference.
- Catch
- Serves models well; does not manage your fleet, routing or tenancy.
LiteLLM
L4 · Inference gateway
A gateway presenting one consistent interface across many hosted model providers and your own self-hosted models.
- Fit
- Applications calling more than one model provider, or planning to.
- Catch
- Adds a network hop, and sits on the critical path of every request.
Kubernetes batch scheduling
L4 · Batch scheduling
Add-ons giving Kubernetes the queueing and gang-scheduling behaviour AI workloads need and plain Kubernetes does not provide.
- Fit
- Organisations already standardised on Kubernetes.
- Catch
- You assemble a scheduler from components rather than buying one.
AWS, Google Cloud and Azure
L3 · Hyperscale cloud
The three large general-purpose clouds, each offering accelerated compute alongside everything else you already run.
- Fit
- Estates where the data, identity and compliance posture already live.
- Catch
- Generally the highest hourly rate, and quota is often the real limit.
RunPod
L3 · On-demand and serverless compute
An on-demand accelerator platform with per-second billing and a serverless mode for inference.
- Fit
- Development, experiments, and inference that scales to zero.
- Catch
- Confirm the guarantees carefully before running anything under a strict SLA.
Other problems
Tell us what you are trying to solve.
Describe your current setup and the problem in your own words. We reply within 2 working days with an honest assessment of whether we can help.