Small vs larger embedding models for cost and accuracy
The current setup runs a small Jina and a Jina in Q8 format inside . The next plan is to stop running the locally and use the instead. That would free local computer resources for larger and s.
The main choice is whether to keep a small 0.6B model or move to a larger 4B quantized model. The intended use is writing and math, so accuracy matters more than simply keeping the setup light.
Key points
- The current setup uses Jina and s in Q8 format.
- The may be moved from local hardware to the .
- Saved local resources could be used for larger and s.
- The decision is between a small 0.6B model and a larger 4B quantized model.
- Writing and math tasks make accuracy the main priority.