Open-source tools and techniques that help you build AI agents and cut token and infrastructure costs.
Claude Design on the macOS desktop app showed a “Failed to update design systems” error when starting a new chat from a design system with existing context. The first design system hit the problem when a third chat was being started. A second design system was created with most of the original assets, and that one worked through a long chat that produced several new designs. Later, the app suggested starting a new chat to reduce about 608,000 tokens of context while keeping the project and files in place. Clicking the new chat button caused the same error on the second design system. The practical failure point was the handoff from a long working chat into a fresh chat tied to the same design system.
The practical problem is how to run embedding work at large scale while improving throughput. The scale is at least 50,000 documents per day, with each document roughly 10 to 20 pages long. This matters in RAG systems because documents must be turned into searchable vectors before an AI agent can retrieve only the parts it needs. The concrete need is for proven operating practices and the current state of the art for making this process faster and more efficient. The item raises the scaling question, but it does not provide specific benchmark results or a finished playbook.
TorchJD is an open-source PyTorch tool for training a model when it has several losses at the same time. Those losses can come from multiple tasks, rules the model must follow, extra helper goals, or regularization. The common approach is to combine all losses into one number, such as an average or a weighted mix, and then train with gradient descent. Another approach is to calculate a separate gradient for each loss and combine those gradients into one update that tries to reduce each loss, not just the average. The second approach can help when the goals conflict with each other, but it usually uses more memory. TorchJD now includes many known methods from both approaches, so teams can test them by changing only a few lines of code. It has also been accepted into the PyTorch ecosystem.
A services company was spending about $4,000 a month on ads and believed many leads were low quality. The bigger problem was the time it took to answer new inquiries. The same inquiry sent through the company’s contact form took 26 hours to get a reply. The same inquiry sent to its main competitor got a human phone call after 11 minutes. The company was not mainly losing to a better competitor; it was losing to a faster one. A Harvard Business Review study found that companies replying within an hour are about 7 times more likely to qualify a lead than companies that wait even a few hours. Many companies take almost two full days to reply at all.
Athena Loops is a lightweight Python tool for coordinating AI agents. It takes one goal, breaks it into smaller tasks, sends those tasks to worker agents, combines the results, and checks them against success criteria. If the result is not good enough, the loop updates the plan and tries again. It can connect to Claude Code, Codex, opencode, aider, and other coding-agent backends through one shared interface. It can run as an MCP server or through a command-line interface. It can run test commands after each worker round and feed failures into the next round. It includes budget guards for iteration count, total time, and agent calls, which helps stop runaway loops. For code changes, it can work inside a separate worktree and save checkpoint commits after each iteration, so partial work is not lost. When using coding-agent CLIs, it can reuse the agent’s existing login session instead of requiring a separate API key.
A 3-month pilot built a conversational AI assistant on top of internal company data. The goal was to let operations and sales teams ask complex questions and get accurate answers. A working demo was ready in the first week, but turning it into something reliable for real work took the next 80-plus days. The main lesson was that the model was not the hard part; the retrieval pipeline and data quality mattered more. An LLM can only answer from the context it receives, so fragmented or outdated text leads to bad answers even with a strong model. About 5% of the work went into connecting the LLM, while about 95% went into data engineering. Company documents looked clean at first, but embedding them exposed several versions of the same client contract across different drives, including draft versions.
Real-time voice AI agents depend on turn-taking more than long answers. They should stop as soon as a person starts speaking, wait until there is real silence, and then answer briefly. Browser microphone permission does not prove that sound is actually flowing. If the operating system has muted the microphone, the audio track can stay silent, voice activity detection may never trigger, and the agent can remain stuck in a listening state. On a server-side Realtime API, muting the local microphone track may not stop billing. Voice activity detection can keep running on the model server, so turn detection must be disabled through a session update to truly pause it. The agent’s own text-to-speech output should not be fed back into speech-to-text, because that can create echo and self-listening loops. In multi-agent setups, an outside orchestrator that controls whose turn it is works better than letting models decide speaking order by themselves.
This desktop AI agent avoids the Computer Use approach, where an AI controls the mouse and keyboard directly. It instead uses an operating-system-based way to connect with desktop apps. The goal is not to support a handful of apps, but to scale to thousands of desktop apps. The team sees Computer Use as too slow and too expensive, especially if each user could require around $300 in token costs. The product focuses on back-office agent processing and aims to run smoothly at high TPS. The idea moved from proof of concept to MVP and then to Early Access after about five months of work on desktop app integration.
Local speech recognition is currently working well with faster-whisper medium or large turbo models on an 8GB VRAM GPU. A useful alternative would need a clearly lower WER than Whisper. It would also need reliable English, Japanese, and Korean support. Speed should be similar to faster-whisper or faster. Accurate timestamps are required. The model also has to run within 8GB VRAM, so very large server-only options would not fit. No specific replacement model or benchmark result is provided.
MetaHarness is an open-source tool that creates a custom AI agent setup from a GitHub repository or a blank project. The generated setup can include a repo-aware command-line tool, a local MCP server, project-scoped memory, skills based on the file layout, governance rules, release checks, and signed proof of origin. It can run in the browser without installation, or from the terminal with the npx metaharness command. It exports settings for Claude Code, OpenAI Codex, pi.dev, Hermes, OpenClaw, RVM, GitHub Copilot, OpenCode, and GitHub Actions. The output can be renamed, branded, and published as an npm package so a team can run the same repo-tuned agent with one npx command. The cost angle is that teams can remove agents, skills, commands, and MCP servers they do not need, leaving a smaller setup that should be faster, cheaper, and easier to understand. It also lets teams adjust model tiers by task, memory scope, and governance rules. Its default safety settings block network access, shell access, and file writes, require approval for risky actions, and add time limits, call limits, and audit logs.
Silia, a custom model design, was used to train a 117 million-parameter model on an H100 in about 5 hours. The computing power was sponsored by Ok-Internal9317. The model files are available on Hugging Face, and the code is available on GitHub. Running it uses uv for inference, with torch, numpy, regex, and colorama installed. The shown sample answer does not clearly answer the question and instead breaks the text into confusing parts, so the model’s practical quality still looks uncertain.
A large set of professional documents needs to be organized, and the documents cover a very wide range of subjects. The goal is to extract an ontology or keyword hierarchy from those documents. The desired result is 3 to 10 broad categories or keywords based on common vocabulary across the collection. No concrete method, tool, or test result is included, so this is an early question about the right approach.
AI agent work can be split across four model tiers instead of sending every request to one powerful model. Most requests first go to a cheap, fast orchestrator that classifies the task and decides whether it needs a stronger model. The main idea is that most agent calls do not need a frontier model; they need quick routing first, then stronger reasoning only when the task is hard enough. Matching the model level to the task level improved both cost and the feel of the agent loop more than choosing one smarter model for everything. Speed was the main bottleneck for interactive agents. When a supervisor takes more than 10 seconds for each decision, the whole agent feels slow even if the answers are good. When the orchestrator decides in 2 to 5 seconds, the loop feels much more usable. The example stack uses DeepSeek V4 Flash as the orchestrator, GLM-5.2 as the primary advisor, GLM-5.2 at maximum effort for deep reasoning, and Opus 4.8 only for sanitized high-stakes work.
Large language models can be tested on coding work by using real merged pull requests from open-source repositories. Each repository is frozen at the point before the human change was merged, then each model tries the same task in a separate container. The score is not based only on whether tests pass. It also checks whether the model’s change has the same effect as the human pull request, whether the code is well made, and how much the run costs. A blinded judge reviews the results without seeing which model produced each answer. The aim is to compare models for code generation and refactoring in a way that is closer to real software work.
Local large language models can slow down on long context because memory movement, not raw computing power, becomes the main limit. During autoregressive decoding, the model must keep reading the Key-Value cache from earlier tokens while it produces the next token. That data has to move from high-bandwidth memory into much faster memory near the processor, and this transfer can make the chip wait. As context grows to thousands of tokens, standard Multi-Head Attention creates heavy memory traffic because each Query head has its own Key/Value partner. Multi-Query Attention reduces that load by letting all Query heads share one Key/Value pair. Newer model designs such as LLaMA 3 and Mistral follow this broader push to get around the memory bottleneck.
Claude Code coordinates several tools to create a product trailer from a live web app. Headless Chrome captures real-time 3D model rotation from the app, while an HTML/JavaScript animation is rendered frame by frame using fixed timing. Audio is generated by code, then ffmpeg combines the visuals and sound into one video. The workflow is designed to produce the same result repeatedly, avoid dropped frames, and make later edits easier. Its cost angle comes from using programmable tools instead of paid video-production steps.
Gemini’s current weakness may be the lack of a stronger long-thinking mode. OpenAI and Anthropic appear to offer higher-effort thinking options for hard tasks, while Gemini seems limited to a level similar to what 2.5 Flash and 2.5 Pro already had by default. The concern is that Gemini 3 added ways to think less and save money, rather than ways to think more and improve hard-task performance. In Antigravity, Gemini 3.5 Flash or 3.5 Pro may finish the same task in 5 to 10 minutes, but with many errors or without truly completing it. By comparison, GPT 5.5 in Codex and Opus 4.8 in Claude Code may spend up to 30 minutes on a task, suggesting a deeper work mode. Deep Think is mentioned as a possible related feature, but it seems different from the higher thinking modes offered by other companies.
Large healthcare policy documents need to be converted from PDF to Markdown before they can be used in a RAG system, but speed and quality are hard to balance. The workload includes thousands of PDFs, with many documents running from 100 to more than 400 pages. Tables matter, some files need OCR, and the documents include English, French, and Spanish. Many files are messy, with rotated pages, scanned pages, mixed layouts, stamps, handwritten notes, and low-quality scans. LLM or VLM processing is not practical because the cost would be too high at this scale. The tool also needs a permissive license such as MIT, Apache, or BSD, because AGPL and GPL tools are not acceptable for a private repository. PyMuPDF is very fast, but it loses too much page layout and table structure. PyMuPDF4LLM gives better results while staying fast, but its AGPL license creates a problem. Docling, used without VLM mode, handles tables and layout much better, but it is much slower on large documents.
The LocalLLaMA community keeps seeing repeated questions about the same large language models, including setup problems and fixes. Reddit search can find some answers, but it is hard to use well, and useful knowledge can disappear as more models arrive. Qwen 3.6 models are given as an example: they can run well, but may need configuration changes and Jinja template updates. The proposed solution is a user-managed wiki that stores model settings, fixes, and troubleshooting notes in one place, with links shared back into Reddit. Hosting resources are available, but the idea needs community and moderator support.
Early customer discovery with enterprise operators points to a need beyond document search or question answering. Conversations with people in asset management and family office real estate showed a similar pattern. Their records, relationships, and recent conversations are spread across many places, so current systems do not keep useful context over time. They want AI to remember past interactions, understand work history, and respond to problems in a way that matches their own judgment. The stronger value may be an AI agent that carries forward hard-won operating knowledge, not a tool that only finds files or answers document questions.
An automation specialist found a little over $40,000 in invoices that were more than 60 days late inside an agency’s operating system. Some invoices were more than 120 days late. The agency was making about $35,000 a month, and the owner was worried about payroll while already-earned money sat uncollected. The reason was emotional, not technical: chasing good clients for payment felt rude. This problem appears common. A QuickBooks report says 56% of small businesses are owed money, the average unpaid amount is $17,500, and nearly half have invoices more than 30 days late.
When many AI agents use the same OpenAI key, the monthly bill can appear as one total number. That makes it hard to see which agent, feature, or run caused the spend. Many tools solve this by sending every model call through a gateway or proxy, but that can add delay, create another failure point, and expose prompts and responses to a third party. The proposed design uses an SDK that wraps the OpenAI client while still sending model calls directly to OpenAI. The SDK measures each call inside the app and sends only metadata to a separate control API, such as token counts, cost, and an agent or run tag. Prompt and response text stays inside the user’s own process. The code change is framed as swapping the OpenAI import, reading the Spaturzu and OpenAI keys from environment variables, and tagging a call with an agent name in one line. The title also says the system supports per-agent budgets and hard caps enforced in the SDK.
A local RAG question-answering system is being built to answer only from a scholar’s biography PDF. The setup uses Ollama, Llama 3.1 8B, Mistral, embeddings, cosine similarity, and BM25. The pipeline splits the PDF into chunks, searches for relevant chunks, sends the top chunks to a language model, and produces an answer. Retrieval seems reasonably good, but the model still invents facts that are not in the document. It also sometimes becomes too cautious and says the text does not clearly state an answer even when the answer is present. The main question is whether the problem comes from retrieval quality, prompting, or the limits of smaller 7B to 8B models for biography-style question answering.
Azeroth Inc., the New York company behind the AI agent platform Jenova, says Jenova has passed $1 million in annual recurring revenue and more than 200,000 user signups across over 70 countries. The company says almost all of that growth came organically. Jenova is built around the idea that AI agents should improve the longer people use them. Conversations, uploaded documents, and user preferences can carry into later tasks instead of disappearing after each session. Memory is presented as the product’s core design, not an optional extra. Pinecone’s vector database supports Jenova’s retrieval layer, and Pinecone published a case study about the setup and its effects. Jenova includes specialized AI agents for tasks such as making manga, researching stocks, and running roleplay games while keeping long-running state across sessions.
Running more than one AI agent at the same time can cause agents to undo each other’s work, ask for the same context again, or repeat work that was already done. A shared coordination layer was built so every agent could read and write the same working record. That record stored decisions, research, open issues, and modules, reaching about 850 notes in one month. The tool’s own meter counted at least about 2 million tokens saved, and it did not count savings it could not measure, so the real number may be higher. Six agents worked in parallel while seeing each other’s work and staying inside their own roles. In one case, one agent was editing requirements while another was deploying; both noticed the overlap and adjusted without human help. The main workflow was simple: one sentence from the human was filed and tagged by an agent, then later agents picked up the topic without rebuilding the same context from scratch.
Several beginner resources were recommended for learning RAG. The main starting point is a YouTube course where a LangChain engineer builds RAG from scratch in Python. The official LangChain RAG tutorial was also recommended; it covers a RAG agent that calls a search tool only when needed and a RAG chain that always retrieves information once before answering. For deeper practice, the list points to LangGraph’s Agentic RAG tutorial, LangChain’s GitHub example repository, and older Reddit discussions about setting up RAG. Extra references include the second edition of “Generative AI with LangChain” and a 2026 AI engineering roadmap article.
Production LLM systems need more than a dashboard that shows total spending by model or provider. The harder problem is finding call patterns that were probably unnecessary. Common waste candidates include repeated routing calls, repeated tagging or classification, tool-selection calls, duplicated context, and requests that become predictable after enough traces. Some calls still need to stay on the best available model, so optimization must separate safe cuts from quality-critical work. Possible approaches include caching, manual rules, cheaper models, LiteLLM, Langfuse, Helicone, semantic caching, evals, or custom tooling. An open-source trace scanner is being built to understand how teams identify these patterns today.
Xiaomi's official Mimo v2.5 blog post explains how the company optimized inference (the process of running a trained AI model to generate answers) to keep prices low while still earning 2-3x profit margins. The poster compares Mimo and DeepSeek V4, noting Mimo does better at coding tasks while DeepSeek V4 has stronger world knowledge. xAI is also pushing low-cost API pricing, but the poster rates Mimo and DeepSeek's approach as better. The hope expressed is that future models will reach much higher capability at the same cost as today's DeepSeek V4, which the poster thinks would already be enough for most everyday use.
aito is a command-line installer that brings together existing tools for reducing token waste in Copilot and Claude Code. AI coding agents can use too many tokens when they reread a whole codebase for every task, place long build logs into the conversation context, or repeat the same explanations in every new session. aito does not create a new saving method from scratch. It combines tools such as OpenSpec, RTK, and ccusage behind one setup flow. `aito setup` installs them with default settings meant to be practical. `aito verify` shows real token counts so users can check the result themselves. The tool supports macOS and Linux and is written in plain Bash. It avoids remote install scripts, telemetry, and a proxy by default, with proxy use only if the user asks for it.
An open handbook is being built to explain how large language models run in real services. The newest chapter covers GPU execution and memory internals. It focuses on why a GPU can sit partly unused during inference, how the memory hierarchy limits throughput, and where the real slowdowns happen. The handbook also covers KV cache, batching, vLLM, SGLang, and TensorRT-LLM, which are all tied to faster and cheaper model serving. Diagrams made with mermaid are used to make the system flow easier to follow. The handbook is still in progress, and feedback is being requested from people who have run inference in production.