Choosing a low-cost online model for a document-based AI agent
A nonprofit climate science organization is building an for its website. The agent is meant to answer only from documents and data that have already been added to its system. It currently works well offline with through Ollama, giving correct information and creating data plots.
The next step is to host it online and possibly expand its scope beyond one watershed. The long-term goal is to offer both an offline version for people without steady internet access and an online version for people who lack a powerful computer or are less concerned about privacy. Traffic is expected to be low because the use case is narrow.
Keeping cost as low as possible is a main requirement. through is being considered, partly because the same may also be useful for research work, though that model may be larger than this task really needs.
Key points
- The project is a website for a nonprofit climate science organization.
- The agent should answer only from the organization’s ingested documents and data.
- running through Ollama already works well offline.
- The plan is to support both offline users and online users.
- Low expected traffic makes a smaller or usage-based online setup more suitable than an oversized model.