A 512MB limit led to a cheaper AI search design

An AI developer tool first used a standard server-heavy design. The backend copied the code repository, split files into smaller pieces, created embeddings, stored vectors, and answered questions. On Render’s free plan with 512MB of memory, one large repository pushed memory use close to 500MB and the service was shut down.

Instead of moving to a bigger server, the design moved work to the user’s browser. The browser now reads the repository, splits files inside a Web Worker, creates embeddings in small batches, and stores vectors locally in IndexedDB. The backend only handles retrieval and LLM inference.

This makes server memory easier to predict, lowers hosting costs, keeps the backend simpler, and improves scaling. The trade-off is that indexes live on each device, so syncing them across devices becomes harder.