Local LLM rig post notes switching Hermes Agent tasks to DeepSeek V4 Flash
A user asked for advice on which model and setup to run on their new local AI server: two RTX 4090 GPUs (48GB combined video memory) plus 144GB of . They currently run Llama.cpp with the Qwen3.6 27B model at Q5_K_M and a 200K-token , but face two problems: only one request can be handled at a time (running two full-length contexts at once would require a technique called RoPE that slows things down too much), and is slow at around 2500 , meaning waits of 40+ seconds before the model even starts responding on long inputs. Generation speed itself is fine at about 75 .
This setup is mainly used for OpenCode, a coding tool, but the user also tried it with Hermes Agent and found accessed via OpenRouter gave better and faster results there instead. Because the machine is shared with others over Tailscale (a remote networking tool), the user wants to support 4 to 8 simultaneous requests, and is considering switching to since its MoE design could let some of the workload be offloaded.
Key points
- Post seeks model recommendations for a with dual RTX 4090s (48GB combined VRAM) and 144GB RAM
- Currently running Qwen3.6 27B via Llama.cpp, limited to one concurrent request and 40+ second delays on long-context prompts
- User found via OpenRouter better and faster than their local model for Hermes Agent tasks
- Server is shared remotely (via Tailscale) with multiple users, so 4-8 concurrent requests is the goal
- Considering a switch to , an MoE model, to help spread out compute load