Can shared home GPUs cut AI answer costs?
The idea is to connect many people’s spare s over the internet and use them as one large AI , similar in spirit to SETI@home. The goal is to let volunteers contribute unused computing power for a useful public purpose. Similar efforts already exist, including Petals, and BOINC could provide a general for sending work to many machines.
The main problem is real-time use. Each step of an AI answer may need communication between different machines, so latency builds up and the answer arrives slowly. Large can raise total throughput, but the user may wait 30 to 60 seconds, or even minutes, for a full response.
Cheap API credits may now cost less than the electricity and complexity of joining a volunteer compute swarm. Distributed inference may be less useful for chat, while dataset generation or distributed training could fit better because those jobs can tolerate delays and may be easier to check.
Key points
- The proposal is to pool volunteer GPUs into a distributed AI .
- Petals is one existing example of a similar distributed AI system.
- is hard because network latency adds up during answer generation.
- can improve throughput, but it can make users wait much longer.
- Cheap API credits may be more economical than volunteer compute once electricity and complexity are counted.