Can a 24GB MacBook run an 8B local model for an app?
The goal is to connect a mobile app to a locally hosted with an . The available machine is a new MacBook Pro Max with 24GB of memory.
The main question is whether an 8B model can run on that machine if it is quantized. The much more expensive alternative is saving money for a DGX AI server.
Key points
- The aim is to connect a mobile app to a .
- The preferred model license is Apache 2.0.
- The available hardware is a MacBook Pro Max with 24GB of memory.
- The key question is whether a quantized 8B model can run well enough.
- A DGX server is mentioned as a much more expensive alternative.