Architecture
Beyond the gateway: the AI control plane.
Every enterprise AI stack starts with a gateway. Most of them, eighteen months in, are quietly rebuilding it as something stricter. A layer that doesn't just route the AI request but governs it, costs it, and freezes a record of what it did. That layer has a name in network architecture, and it's starting to have one in AI architecture: the control plane.
When a gateway is enough
A gateway is the right answer to a smaller problem.
If a team has one or two applications, one or two providers, and the question they need answered is how do we route around an outage and unify the SDK, a gateway, LiteLLM, a thin in-house proxy or a managed routing service, is exactly the right tool.
This page is not an argument that gateways are wrong. It's an argument that they're not the same category of object as what comes next.
The threshold is quiet. It tends to be crossed when:
Past that threshold, a gateway is still doing real work, but it's one capability among several, not the whole story.
The pivot
What an AI gateway can't carry.
A gateway is concerned with how does the request get to the right model? That question is solvable, and a gateway solves it. The questions a control plane has to answer are a category above.
The last row does the real work. A gateway is a thing the team owns and maintains, a tactical fix to a known routing problem. A control plane is a thing the team relies on and reports into, an infrastructure asset with the same posture as a database or a service mesh: load-bearing, governed, and not part of any individual team's surface area.
The argument
Three things a control plane does that a gateway can't.
The ledger is not a log.
Most gateways log requests. Some aggregate the logs and put a dashboard on top, and call that cost visibility. It's observability, not governance. A control plane treats cost the way a financial system treats transactions: every request is attributed to a budget, every budget has a holder, every holder has a threshold, and the threshold is enforced inline before the next request goes through.
The difference between "we logged the spend" and "the spend stopped when the budget was hit" is the difference between a report a finance team has to chase and an infrastructure layer they can rely on.
Compliance is horizontal, not per-app.
When governance lives at the application layer, each team's app implementing its own guardrails, its own audit log, its own access policy, it's per-app paperwork. Every team does it slightly differently; auditors get a different artefact for every system; a policy change requires touching every codebase that ever made an AI call.
A control plane moves governance horizontally. One boundary, one set of rules, one evidence stream. When ISO/IEC 42001 lands as a procurement requirement, the question is "show me the control plane's evidence stream", not "did every app implement the controls."
Performance is a property, not a tuning exercise.
Anything sitting in-path on every AI request is a candidate to be the new bottleneck. A gateway implemented in Python is bound by overhead the runtime cannot eliminate: garbage-collection pauses on the hot path, the GIL, repeated serialisation between middleware layers, network hops in and out of someone else's cloud.
A control plane built as a hot-path Rust process is bound by physics, not by middleware: deterministic sub-2ms overhead at the peak, while Python alternatives spike into seconds exactly when traffic is highest. Performance is a property of the architecture you chose, not a feature you tune.
That's why the verified benchmark, 21,803 RPS at 1.95ms median on a single node, exists not as a boast but as evidence of the architectural posture.
The pivot, once more
Same SDKs. Different category of object underneath.
The upgrade from gateway to control plane is not a rewrite. The application code keeps using the OpenAI SDK, the Anthropic SDK, the Google ADK, whatever it was written against. What changes is what the SDK talks to: not a thin proxy the team maintains, but a strategic layer the team relies on. The translation happens inside the control plane; the application is unaware.
This matters because the most common objection to the upgrade, we already have a gateway, we can't afford to rewrite every app, is misframed. The applications don't change. The boundary they speak to changes. The gateway, in the end, was a tactical implementation of one job a control plane does. The control plane absorbs the gateway and adds the four capabilities the gateway never had.
The teams that are eighteen months into AI in production aren't, in general, arguing about whether the control plane is the right idea. They're arguing about whether to keep building one badly themselves, or to buy one.
See it on a real deployment.
A 20-minute technical walkthrough on infrastructure that looks like yours.