Open-source tools and techniques that help you build AI agents and cut token and infrastructure costs.
A messenger-based AI agent for booking private clinic appointments was shut down after more than half a year of stressful production use. Open-source LLM quality improved a lot over eight months and became usable, but personal use and customer-facing service were very different. PydanticAI made tool calls and API handling easier, but it caused trouble in a system built around synchronous code because its design leaned on asynchronous work. Switching through OpenRouter across GLM, DeepSeek, Mimo, Qwen, ChatGPT, Claude, and Minimax did not remove reliability problems; providers sometimes returned empty answers, and fallback providers could fail at the same time. Structured output also broke: validators could ask the model to fix its answer several times, yet the model could still fail instead of producing the required format. Simple wording or emojis sometimes pushed the bot out of character, caused invented details, or made it ignore system instructions. The riskiest failures involved real actions: the agent booked 11:00 after the user asked for 10:00, then insisted the user had asked for 11:00; in another case it tried to cancel existing appointments to make room for a new one. RAG retrieved wrong services, weak price data confused both the bot and users, addresses were invented instead of fetched with a tool, and delegated agents could fail or encourage made-up replies. Better prompts, multi-agent delegation, guardrails, and newer models fixed many cases, but even 95% correctness was not enough when the remaining failures forced constant monitoring by both the team and clients.
MOTHRAG is an open-source framework for multi-hop retrieval in retrieval-augmented generation. It avoids building a knowledge graph before search. Systems such as GraphRAG, HippoRAG, and RAPTOR can score well, but they depend on an offline graph-building step. When the data changes, that graph often needs a heavy LLM indexing run again. This becomes expensive for data that changes daily, such as prices, company filings, support tickets, or news. MOTHRAG instead uses a graph-free dense index, then coordinates the search steps at query time. New data can be embedded and appended without rebuilding the graph or retraining. The reported benchmark scores were 78.1 on HotpotQA, 76.3 on 2WikiMultiHopQA, and 50.5 on MuSiQue, ahead of GraphRAG, HippoRAG, and RAPTOR in the shared table. The stated cost is about $0.03 per query using ordinary APIs and no GPU. It is not a clear win against GPU-based systems such as NeocorRAG, where the HotpotQA result was roughly tied at 78.1 versus 78.
DeepSeek V4 Flash is being tested in 2-bit, 3-bit, and 4-bit GGUF files. The practical goal is to shrink a large AI model so it can run on personal hardware or smaller servers while lowering memory use and inference cost. Related experiments combine vLLM 0.24.0, llama.cpp branches, DSpark, NVFP4, and quantized KV cache work to test long context and concurrent use. One setup targets two DGX Spark machines with 1.5 million context tokens, a 3 million KV token pool, and 12 concurrent requests, while another checks whether a REAP-pruned NVFP4 DeepSeek V4 Flash setup stays steady at long context on a single Spark machine. A llama.cpp branch has merged fixes for quantized KV cache behavior and compares perplexity against f16 to watch for quality loss. An RTX 5090 MoE setup reports token generation around 22.7 to 21.3 tokens per second and prompt processing around 1105 to 927 tokens per second across prompts from 8,192 to 65,536 tokens. There is still some confusion around model formats, including a case where a GGUF release is described as MXFP4 even though the original model page lists other tensor types, so users need to verify what they are actually running.
openPangu 2.0 Flash is an MoE model trained on Ascend hardware. It has 92 billion total parameters, but only 6 billion active parameters are used when producing an answer. Its context length is 512,000 tokens, and its pretraining used 34 trillion tokens. The model mixes several attention methods to reduce compute, memory use, and memory access cost during long-context inference. SWA handles nearby information, while DSA is used to capture important information spread across a wider context. The model also uses multi-token prediction, with three heads that draft three extra tokens at each step, so answers can be generated faster. After pretraining, it was trained with slow and fast thinking behavior, several reinforcement learning specialists, and on-policy distillation.
Qwen Code is being used less as a tool only for Qwen models and more as a general harness for running open-source coding models. It gives builders a ready place to plug in models such as GLM-5.2 and test coding, review, repair, and tool-use workflows without building a new setup first. GLM-5.2 was released on June 16 and is described as keeping the same broad shape as GLM-5.1, with 744B total parameters and 40B active parameters, while staying at the same listed price of $1.40 per million input tokens and $4.40 per million output tokens. Its reported scores moved sharply: Artificial Analysis Intelligence Index rose from 40 to 51, SWE-bench Pro reached 62.1%, and Terminal-Bench 2.1 improved 17.5% over GLM-5.1. On SWE-rebench, GLM-5.2 scored 51.1% while using 2.62M tokens, placing it near Claude Opus 4.8 xhigh at 56.5% and 2.48M tokens, and above Gemini 3.5 Flash at 49.5% and 1.85M tokens. Small agent app tests compared GLM-5.2 with Kimi K2.7 Code across code, design, and game tasks, including self-review and up to three repair attempts. Another real project test used GLM-5.2 through OpenRouter for a multi-file computer vision app, citing MIT weights, a 1M context, and roughly $1 per million input tokens and $4.20 per million output tokens on that route.
KyroBench is a benchmark for checking whether the context given to an AI agent is safe to use before the agent answers or acts. It goes beyond finding text that looks similar to the question. It checks whether old information loses to newer valid information, deleted memory stays deleted, another customer’s data stays out, prompt injection and misleading near-duplicates are rejected, and returned evidence can be verified. The test cases cover everyday production settings such as support tickets, billing exceptions, legal contracts, incident runbooks, CRM memory, synthetic healthcare records, and coding agent traces. The current official run covers 6 tracks, 36,864 scored checks, 12,288 retrievals per run, and a 1,200 token context budget. The official score is treated as a certification gate, while retrieval signal and latency are shown only as diagnostics. KyroDB, Graphiti/Zep, Qdrant, and Mem0 all scored 0 on official certification. KyroDB had the strongest retrieval signal at 79.1 and complete proof metadata, but missed the hidden semantic and freshness gates. Graphiti/Zep had a 71.6 retrieval signal and the fastest p95 latency at 168 ms, but lacked the proof surface needed for certification; Qdrant acted as a raw vector-store baseline with a 53.3 retrieval signal; Mem0 blocked many strict requests as stale and produced almost no completed retrieval signal.
LLMs often write older, longer code patterns even when modern built-in web features already solve the same job. That matters because output tokens can cost 3 to 5 times more than input tokens in many API price lists. The practical fix is to tell the model, at the start of the session, to use the built-in features available in the runtime, especially in environments such as Deno and Cloudflare Workers. For example, hand-written query parsing can take about 140 tokens, while URL and URLSearchParams can do it in about 12 tokens. Form handling can take 200 to 250 tokens when every field has its own state code, while FormData can collect the form in about 14 tokens. Request timeouts, parallel task handling, and basic interface elements can also be shorter and safer with AbortSignal.timeout, Promise.allSettled, and dialog. A Deno request handler that uses older boilerplate may spend 400 to 600 output tokens before the real business logic starts, while the same handler using built-in features may need only 60 to 90 tokens. Comments also affect model behavior: stale comments can mislead the model, while comments about design intent and constraints give it useful guidance.
An early-stage law firm ran a voice agent for 8 months to handle missed overflow calls and after-hours calls. The agent separated new callers from existing callers, gathered case details, booked or changed consultations, alerted the team about urgent matters, and sent a summary email after each call. It failed at first because callers either asked for a person right away or treated it like an old phone menu, so the conversation flow took about four weeks to fix. After tuning, it handled 571 calls, booked 453 consultations, and completed 96.5% of calls without a human stepping in. About 176 calls, roughly one third, came after hours or on weekends. On real phone calls, latency mattered more than the wording of the prompt because slow replies made callers talk over the agent or hang up. The best intake flow collected one piece of information at a time, such as name, phone, and email, then confirmed everything together before booking. The service cost about $0.13 per minute, making it a weak choice as a company’s main front desk today but a practical replacement for voicemail or after-hours answering.
An agent can start a long task well and then get worse after many steps, often around step ten or later. It may run the same tool again, miss an instruction from the start, or repeat its own earlier reasoning. The model has not changed; the context window has filled up with too much old material. The drop in quality often lines up with deeper token position and a crowded context window. The crowded input usually contains the full raw chat history, early instructions buried under later work, large tool outputs, and the agent’s own past reasoning. A big JSON tool result can add many fields that are never needed again. A better pattern is to summarize old settled turns, keep the decision, remove the raw back-and-forth, and trim tool outputs to only the fields the agent actually needs.
MCP-Dynamic-Router is a gateway that keeps AI agents from reading every MCP tool definition on every turn. The usual MCP setup often loads all connected servers and tool descriptions into Claude’s system prompt. A 200,000-token context window can fit a lot, but sending 100 tool definitions every time can make input token costs much higher than needed. Too many unused tool schemas can also make the model choose tools less reliably and invent wrong arguments for tool calls. Another production problem is that tools cannot be added or removed during a session without reloading the whole context. MCP-Dynamic-Router tries to solve this by showing Claude only the 2 or 3 tools that fit the current job. For voice and chat systems, it can route partial speech while the person is still talking, warm up connections, and prefetch read-only tools. It can also skip heavier routing when a request exactly matches a tool description, and it can return a clarify or no_tool decision instead of forcing a bad tool call.
Google built Paper Assistant Tool, an agentic AI system for checking research papers before formal review. It reads full papers, checks theoretical results, reviews experiments, suggests improvements, and looks for possible flaws. The tool was tested as a pre-submission helper at STOC and ICML, two major computer science conferences, and handled about 10,000 papers with around a 30-minute turnaround. On a math-error test, it found 34% more errors than zero-shot prompting. Its main technique is inference scaling, which means using more model work than a single quick answer to search for deeper issues. Human reviewers still make the final decisions, while the tool is meant to reduce obvious or serious problems before review.
AI automations and AI agents are not the same thing. An automation follows fixed steps, while an AI agent uses a language model to choose what to do next. Mixing up those two ideas can waste money. A three-week agent project could have been handled by a workflow costing about $200 per month. If a team cannot write down the work steps clearly, the workflow needs to be fixed before it is automated. Agents can also fail badly when a task needs judgment. A customer-support agent answered most questions correctly, but one wrong answer said an order was delayed when it was not, which led to a refund and the agent being shut down after six weeks. Lead follow-up is presented as a better current use case because many teams wait hours before responding, and a workflow in n8n can act faster.
Token Warden is an open-source plugin for Claude Code. It watches agent sessions and turns useful patterns into possible memory rules. Each rule is then tested against a fixed test set. A rule is kept only if it saves at least twice as many tokens as it costs to keep in memory. It also must not make the agent fail its tasks. Rules that do not pass those checks are removed automatically. The goal is to stop weak or harmful context-management rules from piling up and raising costs or hurting results.
This is an open-source workflow for reducing how much of Claude’s context window a coding agent uses. The main idea is to avoid sending large parts of the codebase to Claude again and again. Instead, a local graph remembers the code structure and helps fetch only the code slices needed for the current task. In the example given, token use dropped from 12.3 million tokens to 2.4 million tokens. That can let a Claude coding session run longer and lower the cost of repeated model calls. The workflow is available as a GitHub repository, so it can be adapted to other coding agent setups. It is marked as advanced, so it likely needs setup work rather than being a simple plug-and-play tool.
ContextForge helps an LLM handle long tasks without carrying the full conversation history forward at every step. The main idea is to treat the context window as a temporary work area, not as long-term memory. Each step rebuilds a small context that contains only the information needed for the next action. A structured LLM Wiki layer was added so knowledge is stored in a persistent, searchable form instead of flat logs. That made longer runs more consistent. The public repository includes benchmark results for long-running setups, including 180-day and 500-day style tests. The work is also being compared with RecallBench, which checks more than simple information lookup and is meant to test memory behavior across longer tasks.
A multimodal agent handles about 120,000 user interactions per day and needs ongoing checks on production traces. Each interaction can require about 6 judge calls across areas such as faithfulness, helpfulness, safety, tool-call correctness, refusal precision, and staying within scope. Full coverage would mean about 720,000 judge calls per day, or roughly 5 million per week. The current setup samples 16% of live traffic, runs about 800,000 judgments per week, and spends about $2,400 per month using gpt-4o-mini as the judge. Switching to gpt-4.1-nano cut cost by 4 times, but agreement with gpt-4o-mini on a labeled set fell from 87% to 71%. A cascade that used the cheaper model first and sent unclear cases to gpt-4o-mini reduced cost by about 30%, but added delay and proved fragile in operation. Semantic caching reused results for similar prompts and removed about 15% of judge calls, but became hard to manage when scoring rules changed.
The more important fight may be less about which AI model is best and more about who runs the model when a request comes in. OpenRouter recently raised $113 million and now routes 47 trillion tokens each week across different providers. Developers can send requests to OpenRouter, and it chooses where those requests should run. The market is separating into two groups. Fireworks, Together, Groq, and large cloud companies compete on reliability, service promises, and enterprise contracts. Decentralized networks such as Akash, io net, Venice, and c0mpute focus on fewer central limits, such as content filters, account bans, and company-controlled rate limits. DeepSeek now accounts for 4 of the 5 most used models on OpenRouter. When strong models are free or open to use, the infrastructure that runs them becomes a bigger part of cost, access, and control.
JetSpec is a research system for making large language models produce answers faster. It prepares many possible next-token paths in a tree-like shape in one pass, then lets the main model accept only the paths that match its normal output. The reported speedup is up to 9.64x on MATH-500 and 4.58x for open-ended chat, without changing the final answers. With CUDA graph and kernel optimization, the system reaches about 1000 tokens per second on one B200 GPU. Earlier speculative decoding methods often faced a tradeoff: better draft choices became expensive as the tree got deeper, while cheaper draft choices could become inconsistent across branches. JetSpec tries to avoid that by building a causality-preserving tree in a single pass. The project page, code, and blog are public, so others can inspect and test the method.
Token-based billing is pushing companies to look again at small language models. The research describes training a small language model on traces from agent workflows run by frontier models. After supervised fine-tuning, the small model may reach close to frontier-level results while costing far less. The main idea is to avoid calling expensive models through many agent steps every time, and instead teach part of that process to a cheaper model. Real-world results are still an open question.
AI agents that use a computer or browser must find the right buttons, icons, and page areas on screen. Three screen-understanding models, Qwen2.5-VL, UI-TARS-1.5, and GTA1, all scored above 90% on ScreenSpot-v2. Those high scores did not hold up when the tasks were changed in simple, realistic ways. Accuracy fell by 27 to 56 points when the browser zoom was set to 70%, the page style was changed, or the instruction used a relationship such as “the icon above the search bar.” Training UI-TARS-1.5 again on the examples it failed did not fix the problem. The newly trained model performed worse than the original model in every setup. Increasing the training set from 6,500 to 25,000 examples made the drop larger. Both synthetic and real failure examples hurt performance, so the issue is likely the training recipe, not just bad data. ScreenSpot-v2 barely changed during this, meaning a team watching only that benchmark could think a worse model had improved.
Ornith-1.0 is a new open-source family of large language models built for agentic coding work. The released set includes 9B dense models, 35B MoE models, 397B MoE models, and some run-ready compressed versions; the 31B dense model is mentioned in the announcement, but people noticed it was not clearly available in the collection. The team says the 397B model scored 77.5 on Terminal-Bench 2.1 and 82.4 on SWE-Bench Verified, putting it near or above Claude Opus 4.7 on those tests. The 35B model is claimed to beat similar-size Qwen and Gemma models across several coding and agent benchmarks. The 9B model is the most interesting for cost: it is small enough for lighter hardware, yet is claimed to match or beat much larger 31B and 35B models on some coding tests. Its training method lets the model improve not only the answer, but also the step-by-step support structure used to solve the task. Early community reaction is excited, but the real test is whether these numbers hold up in practical agent workloads.
Aura is an AI agent that runs on an Android phone. It includes its own on-device MCP server, which exposes phone actions such as tapping, typing, scrolling, opening apps, moving through the interface, and carrying out tasks. Aura uses these MCP tools to control the phone directly instead of only answering questions. The same server is not limited to Aura. External MCP clients such as Claude Code can connect to it and use the same phone-control tools. The setup is: Android phone, local MCP server, then Aura or another AI agent controlling the device. The project is still in development and experimentation, with a demo video showing it working.
FizzBee is an open source formal verification system that has been under development for several years. A new app built on the same technology turns a rough development prompt into a clearer requirements document. After receiving a prompt, it asks important follow-up questions, turns the requirements into a formal spec, and looks for missing or complicated requirements. It also creates validation scenarios for checking whether the result works as intended. The final output is a specification document that coding agents can follow. Early trials on several projects found that this helped reach working code in fewer rounds of revision.
NVIDIA released Nemotron-TwoTower-30B-A3B-Base-BF16. It is a diffusion-based language model built from the Nemotron 3 Nano 30B-A3B backbone. Most language models write by producing one token after another in order. This model combines a frozen autoregressive context tower with a diffusion denoiser tower, then fills blocks of tokens in parallel through repeated passes. In NVIDIA's default mask-diffusion setup, it kept 98.7% of the aggregate benchmark quality of the autoregressive baseline. Its wall-clock generation throughput reached 2.42 times that baseline.
The open-source RAG infrastructure is meant to plug into existing apps. It can split documents into smaller chunks using different methods for different document types, such as paragraph-based splitting or overlapping windows. For Markdown and HTML, each chunk keeps the heading path so the system knows where the text came from. For spreadsheet-like files, each row-group chunk keeps the column headers so the data still makes sense. It supports many formats, including HTML, Markdown, PDF, text, JSON, XML, DOCX, XLSX, CSV, and PPTX. It can ingest files from a local computer, a web address, or S3. Documents can carry metadata for filtering. Search combines BM25 and vector search with weighted RRF; the current default is 0.7 vector and 0.3 keyword, but that can be changed. It also supports document property filters, such as limiting results to PDFs or documents with a certain tag, and normal queries return chunks with information about the original documents.
Pylon is a full-stack framework meant to make it easier to turn small hobby projects into production-ready apps. Simple projects are often quick to start with React or Next.js, but real services usually need a separate backend, cloud setup, and deployment work, which can become complex and expensive. Pylon combines server-rendered React, TypeScript functions, data entities, access rules, real-time sync, built-in authentication, and background or scheduled jobs. It uses SQLite by default, with an option to move to Postgres for production. Its runtime is a Rust server that runs TypeScript functions and server-rendered React through Bun. The design is inspired by Rails, so it favors clear defaults over many setup choices. One main goal is agent compatibility, meaning coding agents should be able to build and manage apps with less friction.
Anthropic accused Alibaba-linked operators and the Qwen AI lab of trying to pull useful abilities out of Claude. The method at issue is distillation, where answers from a stronger AI model are used to train a smaller or cheaper model. According to Anthropic, the activity ran from April 22 to June 5, 2026, and involved about 25,000 fake accounts and 28.8 million interactions with Claude. The targeted abilities included software engineering, agentic reasoning, and handling long tasks. Anthropic argues this could let rival labs build stronger models without paying the full research and training cost. Alibaba had not publicly responded at the time of the reports.
The chatbot performs well for roughly the first five turns of a conversation, but after that its coherence breaks down. It forgets constraints the user mentioned earlier, contradicts something it said a few messages before, or answers the latest question while ignoring context that was still relevant. It's not a total breakdown, just a gradual drift that makes the conversation feel off. The team had built out evaluation tests using Braintrust and consistently passed their quality thresholds, yet this problem was never caught. The reason: almost every test case was a single prompt-and-response pair, checking only whether the model answered one question correctly. What was actually needed was a way to test whether the model stays coherent across a full 15-20 turn conversation. The team hasn't figured out how to design multi-turn evaluations — whether to build datasets of entire conversations and score them end-to-end, or evaluate each turn individually and then aggregate the results.
A small role-based benchmark compared GPT-5.6's Sol, Luna, and Terra variants across three strategic decision memos, one repository-grounded execution brief, and two bug-fix-with-tests tasks. Scoring covered judgment, grounding, risk awareness, boundary-setting, actionability, and efficiency, and the evaluator could see which model produced each answer, so it was not a blind test. On the multi-layer productization decision, the Fable 5 reference scored 95, Sol max scored 94, Sol xhigh 90, Sol high 87, and GPT-5.5 high 81. On a decision-method review, the reference scored 92 with Sol max close behind at 91. On a bounded-opportunity comparison, Sol high came within one point of the reference (94 vs 95) while using only 81.5 seconds and 369 reasoning tokens; Sol max took 216.9 seconds and 5,178 reasoning tokens without changing the actual decision. On the repository-grounded execution brief, Sol high scored 93 (80.73s, 1,818 tokens) and Sol medium scored 91 (70.66s, 779 tokens) — medium was about 12.5% faster and used 57% fewer reasoning tokens while nearly matching high's quality.
OpenModel is a tool for running and connecting AI models on a local computer or through cloud services. It can pull GGUF model files from Hugging Face, direct model files, and Ollama models, then serve them through local APIs that work like OpenAI or Ollama endpoints. Its dashboard can show request counts, estimated token use, latency, speed, model-by-model usage, and recent requests. With sign-in, it can also show monthly allowance, model price estimates, local-versus-cloud cost comparisons, and usage and cost charts. It can collect token and cost records from cloud-backed coding agents such as Claude Code, Codex, OpenRouter, and BuilderStudio. The records are stored locally as JSONL files unless the user explicitly runs a sync command. It is designed to avoid storing prompts, responses, transcripts, source code, tool arguments, or model weights in the usage store.