Same open model, 8x slower first response depending on backend

Someone building agents on the GLM-5.2 model routes traffic across more than one backend, and benchmarked them because the results surprised them. Using the same model behind the same OpenAI-compatible API, two backends behaved very differently. Under a worst-case cache-busted test (each request forced to skip any cache) with 400-token outputs, a dedicated inference host delivered time-to-first-token (TTFT) of 1.2 seconds, about 77 tokens per second, and streamed cleanly across 167 SSE chunks (small pieces sent one after another).

A major cloud provider's managed endpoint, by contrast, took 10.0 seconds to first token, ran at only about 38 tokens per second, and didn't actually stream at all: stream:true was set on the request, but the server silently built the entire response first, stayed quiet for roughly 10 seconds, then dumped every token at once. For an agent loop, that's a dead 10-second pause every single turn. Moving the request closer geographically only helped 18% and didn't fix the buffering, and testing the dedicated host through the identical code path ruled out any local proxy issue — the problem was on the provider's side.

The important twist: the cache-busted test was the worst case. Real agent traffic reuses a stable system prompt and tool definitions turn after turn, so it's heavily cached. Re-running the same test with a warm cached prefix dropped the buffered endpoint's TTFT from 10 seconds to 2.7 seconds and mostly eliminated the buffering — meaning the scary 8x gap was largely a cold-cache artifact, not something a production agent would normally hit.