Can several Apple Silicon machines run one LLM server together?
Four MacBook Air M2 machines with 16 GB of memory each are being considered as a small cluster for inference. The planned setup would connect the machines through in a ring, so they can work as distributed nodes. An earlier test with exo-explore/exo was slow, producing under 6 to 8 on a Qwen 2.5/3.5 9B 4-bit model.
The next idea is to use vllm-metal with separate roles: one node handles the early input work, and another handles answer generation. LMCache would be added so the M2 machines can act as shared or offloaded KV cache storage. The open question is whether vllm-metal can already run or across multiple machines in a stable way.
Current notes suggest cross-Mac communication through mx.distributed is still experimental or still being built. It is also unclear whether LMCache can plug into vllm-metal well enough to make a useful distributed KV cache layer and improve speed.
Key points
- The setup uses four M2 MacBook Air machines with 16 GB of memory each.
- A test with exo-explore/exo stayed below 6 to 8 on a Qwen 9B 4-bit model.
- The proposed design splits early input processing and answer generation across vllm-metal nodes.
- LMCache is being considered for shared or offloaded KV cache storage.
- Multi-machine inference still appears experimental.