A fully offline RAG setup cuts cloud use for personal documents
A personal book and PDF was turned into an offline RAG system that runs only on one local machine. The goal was to stop paying recurring OpenAI bills just to search and summarize private documents. The machine was a Corsair AI Workstation 300 with and 128GB of .
That memory setup made it possible to keep a 122B model, an embedder, and a reranker loaded at the same time. This reduced the need to unload one part of the system before loading another. The build avoided and LlamaIndex and used Ollama with custom code instead.
The difficult part was AMD ROCm, which caused setup and error-fixing problems that took a lot of time.
Key points
- The system keeps private books and PDFs off the cloud.
- The main reason was to reduce recurring usage bills.
- 128GB of allowed several large AI parts to stay loaded together.
- The build used Ollama and custom code instead of or LlamaIndex.
- ROCm setup was a major source of friction.