Picking an internal AI gateway for multiple teams: a hands-on comparison
This describes a search for an internal-only AI gateway (for , POCs, and demos — not production or customer-facing use) that lets several teams share access to models from OpenAI, Anthropic, Bedrock, and others. Three drive the choice: via so raw vendor keys aren't handed out directly, enough usage visibility to see which team is spending what, and low enough maintenance that it doesn't need a dedicated engineer. Comparing the options: LiteLLM stands out as the most natural starting point, thanks to an active community and quick setup with little upfront overhead.
TrueFoundry comes up often when governance matters, and its per-team cost tracking is noted as genuinely strong. Portkey looks more polished out of the box, but there's hesitation about whether its self-hosted option is as capable as its managed version, which matters since is a requirement here. Kong is powerful but considered overkill for what is essentially an internal proxy needing just keys and dashboards.
OpenRouter is also mentioned, though the excerpt cuts off before further detail.
Key points
- Scope is strictly internal (/POCs/demos), not production or customer-facing
- Core : virtual-key , per-team usage visibility, low maintenance overhead
- LiteLLM: active community, fast to set up
- TrueFoundry: strong governance and per-team cost tracking
- Portkey is more polished but support is a relative unknown