Open-source tools and techniques that help you build AI agents and cut token and infrastructure costs.
Real-time voice agents need a tougher speech recognition test than clean recordings and one final WER score. A useful public test set would include speakerphone calls from a moving car, customers who say a number and then correct it, and quiet moments filled with fan noise, TV sound, keyboard clicks, or hold music. It should also test two people talking over each other, hard addresses and local names, long meetings where timestamps and speaker labels slowly drift, and code-switching with English product names inside another language. Other cases include angry fast speech, a quiet speaker after a loud speaker, and a full agent workflow where the transcript feeds intent detection, CRM or calendar fields, and a summary. The suggested metrics include WER, entity error rate, time to first usable text, time until the transcript stops changing, partial rewrite count, diarization drift, timestamp drift, false speech detection during silence, p95 latency, and workflow success rate. Smallest AI Pulse is named as one API worth testing specifically as the live STT layer for real-time voice agents, not just as a generic transcription tool.
CertLocker is a DevOps control plane for secrets and tokens, with a focus here on safer MCP agent access. Claude and MCP agents can be used for real client work such as reports, scripts, standard procedures, campaign checks, and small automation tasks. The access problem appears when staff and contractors in different countries all need credentials to do their work. A single shared .env file can end up holding AI provider keys, client advertising tokens, Search Console OAuth access, webhook secrets, GitHub tokens, and procedure repository access. Once that file is shared, a contractor working on Client A may also be able to see Client B credentials. An agent checking procedures may also be able to read webhook secrets or tokens that it does not need. Scoped MCP tokens aim to give each person or agent only the access needed for a specific job.
In an SRE and security operations AI agent project, Strands was used first and was very easy to start with on AWS. The project later moved to LangGraph because more detailed control was needed. The agent’s job was to read logs and investigate issues. After one month of testing, the results were almost the same when the same models were used. The main difference was the surrounding framework and workflow, not the model output itself.
A local comparison of two models on the same agent work found that a model that thinks in fewer tokens can answer faster than one that generates tokens more quickly. Qwen3.6-27B used an average of 1,856 thinking tokens, while ThinkingCap-27B used 675. Both got 10 of 12 checkable answers correct. Average response time fell from 30.1 seconds to 17.8 seconds, and long reasoning requests were up to 3.2 times faster. ThinkingCap-27B decoded more slowly, but it finished sooner because it produced far fewer tokens overall. On a separate tool-call test, tool-name accuracy rose from 0.089 to 0.179. The tradeoff was weaker built-in knowledge: a lore-recall score dropped from 0.236 to 0.194, so agents that depend on model memory instead of retrieval may lose quality.
An AI agent or background job can stop halfway through a task and then run again. That retry can cause the same real-world action to happen twice, such as charging a customer twice, sending the same email twice, or writing conflicting data. Idempotency keys reduce duplicate work inside one provider, but they do not fully solve cases where a call times out and the result is unknown. They also do not cover a workflow that spans several systems, such as Stripe, email, and a database write. A small open-source npm package was built to wrap risky calls and track their state through steps like intended, executing, committed, and verified. After a timeout, it checks the real provider status before deciding whether to retry.
When AI agents connect to MCP servers and internal APIs, many setups still depend on shared credentials or long-lived credentials stored in environment variables. That makes it hard to cut off only one agent if it starts doing something wrong. Teams may instead have to rotate a shared credential and restart services. After an incident, they also need a reliable way to reconstruct what the agent accessed and what actions it took. From an IAM viewpoint, separate access control, least privilege, session termination, and audit logs are normal expectations, and the open question is how production AI agent systems are meeting those expectations today.
Baseline is a beta tool for reducing guesswork when improving prompts for AI agents. Small wording changes can look better after a few sample answers, but they may quietly make other tasks worse. The tool treats prompts like software that needs regression testing. Users write a plain-language rubric with weights for what matters, such as accuracy, tone, and whether the answer resolves the task. The optimization engine runs the prompt across an eval dataset, scores the outputs, rewrites the instructions, and scores them again until it finds a stronger version. In the example shown, an output score rose from 0.62 to 0.94. The beta is limited and aimed at people actively building and testing AI agents, with access codes sent by request.
Conduit is a local gateway that lets many AI apps share the same MCP tool servers. Claude, Cursor, Codex, and similar clients can point to Conduit instead of each app being configured separately. Its main cost-saving feature is lazy discovery. Instead of sending every tool description to the AI on every request, Conduit first exposes only three meta-tools for status, search, and tool calls. In the example given, 3 MCP servers with 62 tools would add about 24,000 tokens before any real question is asked, while Conduit lowers that to about 660 tokens. Its benchmark claims up to 91% fewer total tokens at the same task success rate, 97% less tool-description overhead per request, and 99.6% less overhead on a real 415-tool catalog. Secrets are stored in the OS keychain rather than client config files, and Conduit also adds tool-change detection, quarantine warnings, prompt injection detection, tool toggles, and call logs.
An internal research agent needed about 40 to 60 large language model calls for each task. Using GPT-4o for all of them cost more than $1 per task, even though many calls were only for routing or extracting information. Kimi K2.7, GLM 5.2, Qwen 3.7 max, and DeepSeek V4-Pro were tested on the same evaluation set, with about 300 runs each, using standard OpenAI-style tool prompts and no model-specific tuning. Kimi K2.7 handled tool calling well, including parallel tool use and following the required schema. GLM 5.2 stayed usable when given very long context of about 60,000 to 80,000 tokens. Qwen 3.7 sometimes returned tool arguments as a JSON string instead of an object when under load, which a retry wrapper could fix. Deeply nested schemas with optional fields caused more trouble for every open-source model than for GPT-4o, so flatter schemas worked better. The final setup used Kimi for tool steps, GLM for synthesis, and GPT-4o only for the final answer shown to the user, cutting cost from about $1.10 to about $0.35 per task.
A team used GPT-4o as an automated judge for every deploy. The setup scored prompt changes against an 8-part checklist, and it worked for the first three months by catching a few clear regressions. Later, the machine learning lead changed a system prompt to improve one edge case, and the judge gave the change an 8.7 out of 10. After release, about 3% of users hit a different flow that produced the wrong output format. The judge had never seen that format in its examples, so it treated the change as fine. The deeper problem was split tracking: product prompts were stored in PromptLayer, while the judge prompt lived in a Notion document. That let the judge drift between deploys without anyone seeing it. Putting both prompts in the same versioned system made the drift visible before release, though automated judges still struggle to catch subtle quality problems.
Conduit is a local gateway for managing many MCP servers in one place. AI clients such as Claude, Cursor, VS Code, and Codex can connect to Conduit instead of each being configured separately. Its main cost-saving idea is to avoid sending every tool description to the AI model on every request. With 3 MCP servers and 62 tools, tool descriptions alone can use about 24,000 tokens before the user asks anything; Conduit reduces that to 3 meta-tools and about 660 tokens. Its benchmark reports up to 91% fewer total tokens while keeping the same task success rate. It also reports 97% less tool-description overhead per request, and 99.6% less overhead with a real 415-tool catalog. Secrets stay in the operating system keychain instead of being sent to a cloud service. It also includes controls for limiting which tools each agent can use, hiding risky tools, logging tool calls, detecting changed tool definitions, and marking suspicious outside content that may try to steer the agent.
A company security review found AI agents that had been built, deployed, and then left behind after their creators moved to other teams. Some had no owner, no documentation, and still had access to live production systems. The urgent problem was that basic questions had no clear answers. The company could not easily tell which AI agents could reach customer data, which ones were still being used, which ones were abandoned, or who should be called if something broke overnight. A useful agent registry needs four parts: a searchable catalog, access control, execution logs, and usage metrics. If any one part is missing, the registry becomes little more than a cleaner-looking spreadsheet. The team first built a temporary internal version, then reviewed vendors, and chose TrueFoundry because it covered those four needs without more custom maintenance.
A small legal-tech team is trying to attach legal support to more than 2,700 real questions from an official government notary exam. Each question has a marked correct answer, but it does not include the article, case law, or other legal basis that proves why the answer is correct. The goal is to help exam candidates understand the reason behind the answer, not just memorize it. Manually matching all questions to the right legal text would take too much work, so the proposed system uses RAG to search a national legal code and suggest supporting sources automatically. A human would review the results later before they are trusted. The proposed stack uses Supabase and pgvector, Cohere’s legal embedding model, hybrid BM25 and kNN retrieval, and a high-end LLM to generate explanations or citations from the retrieved material. The larger product is a study platform that finds each candidate’s weak legal topics and gives targeted practice instead of generic flashcards.
A tester checked whether AI agent memory tools truly forget a fact once it's corrected, and found many do not. The easy case everyone handles: you say the region is Frankfurt, then correct it to Ohio — a good memory system should answer Ohio afterward. The harder case almost nobody checks: what happens if the old value gets mentioned again later, not as an attack, just a user repeating a preference they forgot they'd changed, or one stray old line buried in a long chat. Does the dead value, Frankfurt, come back? There's a concrete reason it can: cosine similarity, the common method for comparing meaning between texts, is bad at telling a contradiction apart from a simple duplicate — one cited paper measured this at an AUROC of about 0.59, barely better than a coin flip. A corrected value often reads as more similar to the original than an ordinary rephrase would, so a similarity-based memory store has no clean way to tell that the repeated old value is actually a dead fact resurfacing. This isn't just theoretical: memory poisoning is an active 2026 security topic, listed by OWASP, and an attack called MINJA reportedly succeeds at injecting an agent's memory 98.2% of the time. To measure this directly, the author built a small benchmark: state a fact, correct it, then restate the old value, scoring at the answer level with 30 test cases run locally. The metric, echo-resistance, measures how well the corrected value sticks, where 1.0 is ideal. The author's own simple keyed memory store scored 0.00 with safety checks turned off — a complete failure, because the restatement was the most recent write and simply overwrote the correction.
OpenAI has started Patch the Planet, a program that helps open-source software projects find and fix security weaknesses. The program was created with security firm Trail of Bits, with HackerOne and Calif also involved. Open-source maintainers get free security consulting, help finding vulnerabilities, help writing fixes, codebase improvements, and support for adding AI security tools to their normal work. OpenAI also announced a new limited-access version of GPT-5.5-Cyber and expanded its Codex Security scanner as an app plug-in. GPT-5.5-Cyber scored 85.6% on the CyberGym benchmark, above Anthropic’s Mythos 5 score of 83.8%. Patch the Planet participants receive six months of free ChatGPT Pro and six months of Codex Security, plus help improving infrastructure and workflows. The program has already worked with more than 30 open-source projects and found hundreds of bugs. Trail of Bits is spending major effort not only on finding bugs, but also on customizing agents for each codebase so maintainers can keep using them afterward.
DeepSWE is a new benchmark for checking how well advanced AI coding agents perform on real software engineering tasks. Its tasks are written from scratch, instead of being adapted from existing commits or pull requests, which lowers the chance that a model already saw the answer during training. The tasks cover 91 repositories across 5 programming languages. The prompts are about half as long as SWE-bench Pro prompts, but the solutions require 5.5 times more code and about 2 times more output tokens. The tests are hand-written to check whether the software behaves correctly, not whether the code follows one expected implementation. DeepSWE is open-source and available on GitHub.
Automated evaluations are becoming common for AI changes, but many teams still do manual checks before approving pull requests. Those checks can include traces, prompts, retrieval changes, and production metrics. The main question is whether passing tests gives enough confidence that the AI change is safe for real users. Automated evaluations may not show every issue in how an answer was produced, how outside information was found, or how the change behaves after release. Teams still need to decide what humans should review when the automated checks pass.
AI feature changes can be harder to judge than ordinary software changes. When prompts, models, RAG pipelines, agent workflows, or tool integrations change, passing CI and automated evals may still not prove the change is safe for production. The decision depends on what evidence reviewers inspect before approving a pull request and which systems they check, such as GitHub, Langfuse, LangSmith, Promptfoo, dashboards, and logs. The hardest risk is that a change can pass tests but behave differently with real users. The practical issue is the lack of a clear standard workflow for reviewing AI changes before deployment.
Wisp is a free, MIT-licensed open-source app that lets people call their chosen LLM without leaving the app they are using. A small overlay opens with a hotkey, an action can be picked quickly, and the answer appears there as it is generated. It can use selected text, the clipboard, the active app, a browser, a document, or a screen snip as context, and each source can be chosen from the overlay. It can rewrite text and paste the result back into the original field. It also supports voice questions, direct dictation, and local or optional cloud TTS. It works with hosted model providers and local OpenAI-compatible endpoints such as Ollama and LM Studio. It includes MCP client and context-server support, a full chat window for longer conversations, and isolated Python add-ons. It has no project-run backend or hosted storage layer; settings, chats, and optional memory stay on the user’s machine, while API keys are kept in the operating system keychain.
JAS RAG is a small local tool for testing retrieval-augmented generation before adding heavier parts. It indexes Markdown files, splits them into documents and chunks, and stores everything in SQLite. It uses SQLite FTS5 for keyword search, searches at the chunk level, and formats the matched chunks as context for a large language model. It does not need embeddings, a vector database, rerankers, hosted services, or API keys. The workflow is to index a small folder, run a few questions, check whether the top 1 and top 3 results are right, and inspect the returned chunks directly. Vector search is added only if the test shows SQLite FTS5 is not good enough. It is not a full RAG platform or production framework, but a simple local baseline for checking retrieval quality.
AI agents are starting to act for people across more apps and services, so identity and accountability are becoming more important. The goal should not be to block all automation. Platforms need a way to separate legitimate agent activity from harmful use. They also need to know what an agent is allowed to do and what permissions it has. When risky behavior is detected, the platform should be able to verify the real person responsible for that agent.
OpenScience is an open-source AI workbench for scientific research. A person gives it a goal, and it can move through a research workflow: finding papers, forming a hypothesis, writing and running code, running experiments, analyzing results, and writing up what happened. It includes a default research agent plus specialist agents for biology, physics, and machine learning, along with helper agents for review and literature checks. It includes more than 290 skill areas, covering model training, evaluation, datasets, molecular and clinical biology, chemistry tools, papers, figures, and cloud computing. It can query scientific databases such as UniProt, PDB, Ensembl, ChEMBL, PubChem, arXiv, OpenAlex, and Semantic Scholar. The browser workspace includes a file tree, editor, terminal, session history, and views for molecules, genomes, structures, and plots. It works with models from Anthropic, OpenAI, Google, and many other providers, using the user’s own API key or Synthetic Sciences’ Atlas managed model wallet. OpenScience runs as a local server, but its agent is not sandboxed, so sensitive work may need a container or virtual machine for stronger isolation.
FastMCP 3 can be used to build a production-ready MCP server. The setup includes JWT authentication, hiding tools by user group, audit logs, and S3 signed URLs for file access. JWT authentication limits use to approved people or systems. Hiding tools by user group means different users only see the actions they are allowed to use. Audit logs keep a record of what happened. S3 signed URLs let files be shared through controlled links instead of making them openly available.
A CLAUDE.md file was made as a rule set for using an AI agent in a more predictable way. It is built around two ideas: explicitness and continuity. Explicitness means the agent’s behavior, duties, checks, and handoffs should be written down clearly. Continuity means project context should survive across sessions and delegated tasks, instead of being rebuilt each time. The guidance covers project knowledge management, model routing, worktree use, verification, task records, and multi-agent handoff steps. The goal is to test it in real workflows and learn what helps, what feels excessive, and what should be improved.
Keeping an AI agent reliable in real service often needs more than one tool. LangSmith, Langfuse, and Arize Phoenix are common choices for tracking what happened during a run and checking output quality. Tools that can stop a risky action before it happens are less common, so teams often add separate guardrail libraries such as LLM Guard, NeMo, or Lakera. Model routing, backup routing when a model fails, and limits on which tools a model may use often sit in another layer, with LiteLLM used as one example. A typical setup becomes one tool for traces and evaluations, one guardrail library, and one gateway in front. The weak point is that these tools often do not share one run ID, so debugging can mean matching the same failed run across different dashboards by time.
The RAG project already includes several advanced search features. It combines BM25, which finds matching words, with vector search, which finds text with similar meaning, and then blends the results with Reciprocal Rank Fusion. It uses Cohere reranking to reorder results and HyDE query expansion to make the user’s question easier to match with useful documents. It keeps a saved search index, updates only new or changed content, adds source citations, and includes a no hallucination policy. The main issue is what else is needed before this can be considered production-ready, instead of just a more polished version of a basic RAG demo.
An AI agent may return several resources from a system after completing a task. Each resource can have an internal ID, but that ID may not be allowed in the visible answer shown to the user. If the user later says “show me more about #3,” the agent still needs to know which real resource that number refers to. The main issue is how to keep the visible list number, the hidden internal ID, and the real resource connected across several turns of conversation. The example flow is: the user asks for their top X items, receives a list, then asks for more detail about item 3.
A chatbot needs to use backend data that is refreshed through APIs every 5 minutes. The dataset is large, so sending all of it to the LLM on every request would use too many tokens and raise cost. A standard RAG setup may also be awkward when the knowledge changes this often, because results can become stale or need frequent rebuilding. A better architecture would need to fetch only the small slice of data needed for each question, possibly using database search, vector search, caching, tool calling, and query planning together. The core challenge is making frequently changing enterprise data usable without overloading the model input.
A recurring issue over several months is that a large language model may say it will use a tool in one way, then call the tool in a different way. The final effect may be similar, but the stated method and the actual action do not match. In one example, GLM said it would use a transform approach to edit a buffer, which meant writing a small Lisp-like expression for the edit. Instead, it wrote a diff and applied that change. The same kind of mismatch has also been seen with Claude Sonnet and Claude Opus, so it does not appear to be limited to GLM.
The focus is on automations, AI agents, and repeating work flows that keep running in real work or personal life. The examples should exist in a codebase or on GitHub, not just as short-lived demos. The useful areas can include machine learning, large language models, scripts, deployment automation, developer tools, home automation, and knowledge management. The main questions are what was built, what problem it solves, and whether it stayed useful over time.