Practical limits and setup lessons for running Hermes Agent locally
The local AI setup uses Ubuntu, an RTX 3090 with 24GB of VRAM, Docker, Ollama, Open WebUI, , and . Qwen and gpt-oss models have been tested, and LM Studio is being added as a second way to run models. Local use can keep off outside services and avoid usage-based API charges.
It also requires direct choices about models, , memory limits, usable , file paths inside and outside containers, and where model and app data are stored. needs clear permissions and strict boundaries around where it may write files. Logs, model loading, and recovery from failures also need to be managed.
Local operation still costs money for hardware and electricity and takes time to set up, while 24GB of VRAM limits which models can remain fully loaded on the . remain useful for harder reasoning, longer context, or models that cannot run on this machine. The goal is choice rather than replacing every : private or repeatable work can stay local, while demanding work can use the cloud.
Key points
- The setup combines and with Ubuntu, an RTX 3090, Docker, Ollama, and Open WebUI.
- Choose the model and level by balancing the 24GB VRAM limit against the the task needs.
- Separate host and container file paths, and make model and application data persistent across restarts.
- Restrict 's permissions and clearly define where it is allowed to write files.
- Prepare logging and failure recovery, and use when local reasoning power or is insufficient.