Choosing an open-source model for a local RAG app
A hackathon team wants to build a RAG app that runs only on two laptops instead of using online servers. One laptop has a 13th generation i7 processor and 32GB of memory, but no . The other has a 12th generation i5 processor, 16GB of memory, and a 4GB Nvidia .
The goal is to answer questions using retrieved information, keep acceptable, and reduce wrong but confident answers. The team is trying to choose which fits those limits best.
Key points
- The goal is a local RAG app for a hackathon.
- The available hardware is one 32GB laptop without a GPU and one 16GB laptop with a 4GB Nvidia GPU.
- The team cares about speed and fewer wrong answers.
- Local may reduce , but hardware limits matter.