Choosing a self-hosted LLM gateway for real production use
A production needs , routing across multiple s, cost tracking by team, strong , and low maintenance work. LiteLLM, Portkey, and TrueFoundry are the main options being considered.
LiteLLM appears to be the default choice for many teams, but upgrade stability concerns keep coming up. Portkey looks polished, but its setup seems less convincing.
TrueFoundry is often mentioned when teams need governance features, but there is little visible experience from teams using it beyond a short . The practical question is which gateway is actually holding up under real traffic, and which one teams would choose again from scratch.
Key points
- A production should support , multi-, team cost tracking, and .
- LiteLLM looks like a common default, but upgrade stability is a concern.
- Portkey appears polished, but its story may be thin.
- TrueFoundry may fit teams that need governance, but public production experience seems limited.
- For AI agent cost reduction, routing and cost tracking are the most important parts to evaluate.
Sources covering this story (5)
- r/LLMDevsChoosing a self-hosted LLM gateway for real production use ↗
- r/LLMDevsBest LLM gateway for internal access across multiple vendors, LiteLLM or something else? ↗
- r/LLMDevsWhat LLM gateway is actually worth it in 2026? ↗
- r/LLMDevsWhat's everyone actually using for an AI gateway in prod? Tired of duct-taping LiteLLM together ↗
- r/LLMDevsWhich MCP gateway is actually being used in production? ↗