For local RAG, prefill speed can matter more than answer speed
In local RAG, the main slowdown can be prefill, not decode. RAG adds thousands of tokens of retrieved material to each prompt, so the model must read a before it can start answering.
On machines such as , decode can still look fine on some at about 25 to 40 , but prefill can be much slower than on a discrete GPU. A single 24GB discrete GPU may process the same in a few seconds, while a setup can make the user wait 20 to 60 seconds before the first token appears.
can still be fine for where waiting is less painful. For interactive RAG, a budget build should leave a free PCIe slot so a discrete GPU can be added later to handle prefill.
Key points
- RAG often sends thousands of retrieved tokens into every prompt.
- Slow prefill can create a 20 to 60 second wait before any answer starts.
- Decode speed of 25 to 40 can still feel slow if prefill is weak.
- A 24GB discrete GPU can handle much faster than some setups.
- A budget system should keep a free PCIe slot for adding a discrete GPU later.