Deploying an Ollama-based RAG app without burning local GPU
A RAG app has been built with running through Ollama. The is granite-8b, the is BGE-M3, and the is qdrant. The whole setup, including the qdrant data, currently lives on one personal machine.
The main deployment questions are what tech stack to use, how other people would get answers if the model stays on local hardware, and whether the model must be uploaded to to create an endpoint. A key concern is avoiding a setup where other people’s usage consumes the owner’s GPU heavily.
Key points
- The app uses granite-8b, BGE-M3, and qdrant.
- All models and stored data are currently on a personal machine.
- The owner is unsure how outside users would receive answers from a .
- endpoints are being considered but the reason for using one is unclear.
- The core issue is how to deploy apps without uncontrolled GPU use.