A sharing standard for llama-server setups on 24GB GPUs
The goal is to collect launch settings that have worked well on 24GB s such as the RTX 3090, Radeon RX 7900 XTX, and . Each setup should use as much as practical while providing a KV cache for at least 200,000 tokens. The launch command should include the computer's , because RAM capacity can affect cache speed and whether the setup works at all.
and CPU details should also be included. These parts of the computer can change how much system and can use.
Key points
- The target cards have 24GB of : RTX 3090, Radeon RX 7900 XTX, and .
- Shared settings should use as much as practical.
- Each setup should support a KV cache of at least 200,000 tokens.
- should be listed beside the launch command.
- and CPU details should also be recorded.