Big context can make local AI agents slow
can slow down sharply when a conversation becomes long. Even on an M5 MacBook Pro with 128GB of , a context around 16,000 tokens became a serious bottleneck in this experience. The main issue is the growing context, not just the model choice.
A more workable pattern is to split a job into tiny parts, run each part in a fresh short session, and pass only the short result or summary to the next step. For an overnight scraping job that becomes a morning dashboard, many small workers can each extract one piece of information, then one aggregator can combine only the short summaries. This map-reduce style fits better than one agent carrying a long history.
The open question is which agent frameworks support this stateless worker pattern well, since tools like CrewAI, AutoGen, and default LangChain often carry too much history forward.
Key points
- Long can make local model inference very slow.
- A context around 16,000 tokens can already become a bottleneck for work.
- Splitting work into small fresh sessions keeps each model call shorter.
- A map-reduce workflow uses many small workers and one final aggregator.
- CrewAI, AutoGen, and LangChain may need careful setup if they carry full history by default.