Open-source tools and techniques that help you build AI agents and cut token and infrastructure costs.
Tool-using large language model agents can be tricked when they read outside information from web pages, files, or application programming interfaces. This risk is called prompt injection, and it becomes more serious when an agent can take real actions after reading that outside information. Sentinel Gateway is a middleware layer that keeps trusted commands separate from untrusted outside data. The main idea is not to guess whether an input is harmful, but to require a signed authorization token before any execution step can happen. This separates what the agent observes from what it is allowed to do. The implementation includes FastAPI middleware for tool calls, token checks for execution requests, a Streamlit screen for review and debugging, logs of agent choices and tool use, and either local storage or Postgres storage. It is also designed to fit multi-agent setups, including Claude-based sessions.
A coding agent can use an architecture diagram as a real checking tool, not just a pretty document. The useful setup is Event Storming, where the system is laid out as a timeline of cause and effect. An event leads to a reaction, the reaction creates a command, and the command creates another event. This gives every part a rule: it should have a clear cause or result. A simple graph check can then inspect the cause-to-effect links without asking the model to make a vague judgment. It can return clear gaps, such as an event with no cause, a command with no resulting event, a policy that connects nothing, or a card that sits alone. That gives the agent a feedback loop based on fixed rules instead of tired or overconfident guessing.
When choosing a vector database for a RAG workload, a QPS ranking can hide the real operating cost. A setup may look strong when only response speed is measured, but the result can change once metadata filtering, returning extra stored data, frequent new inserts, or separate tenants and namespaces are included. Costs can also shift depending on whether traffic comes in bursts or stays steady. VDBBench is useful because it frames the choice around workload tradeoffs instead of a simple winner list. The key question becomes not only which database is fastest, but which one is fast enough for a specific usage pattern at an acceptable cost. Tests such as how quickly new data becomes searchable, delay after idle periods, search with returned payloads, and multitenant search are closer to real production use than a fixed query-only test.
PatSnap has open-sourced two tools it built for handling patent documents at large scale. Hiro-Smart-Doc is a self-hosted FastAPI pipeline that takes PDFs, Office files, and images, finds the layout of each page, then outputs content in the right reading order as Markdown. It uses RT-DETR to detect 25 kinds of page regions, including layouts with multiple columns. Tables are exported as HTML, formulas as LaTeX, and normal text as Markdown. The text recognition layer uses Hiro-MOSS-OCR, a 0.3B OCR model trained from the ground up on more than 50 million technical documents. The model scores 93.63 on OmniDocBench v1.5 and is reported to process 58 queries per second on one RTX 4090 with vLLM. Both tools are available under the Apache-2.0 license.
Awesome Agent Evals is a public repository for people building and evaluating AI agents. It collects papers, blog posts, talks, courses, tools, and benchmarks, and adds short notes on what each resource covers and why it matters. It includes more than 443 selected links and 146 deeper reading notes. The list was built from a large citation crawl across 11,600 papers, targeted discovery of practitioner writing, notes from 47 talks and podcasts, and checks for gaps in each section. The starter set points readers to resources on agent evaluation, LLM-as-judge, error analysis, cost as an evaluation metric, benchmark limits, and trusted third-party evaluations. A companion playbook gives practical examples for judging AI outputs, analyzing mistakes, grading agent steps, checking real state changes, and using CI to block weak changes.
A firsthand Odin game development knowledge base was built and maintained with MiniMax-M3 inside Kilo Code. The setup combines repeatable Python scrapers, a curated local knowledge base, and a subagent that reads only selected files instead of pulling in everything. The scrapers collect material from a Skool membership area, odin-lang.org documentation, awesome-odin, and Zylinski sources found through RSS, while keeping the content on the local disk. The subagent reads INDEX.md first, chooses 2 or 3 relevant knowledge base files, and cites exact file paths and line numbers. The .kilo/skills folder contains six task tools for knowledge base search, code formatting, scraper control, Odin pattern finding, daily planning, and Pyright checks. The official documentation folder stores 11 pages from odin-lang.org/docs with attribution, and the planning folder follows a strict day-by-day template that is not manually edited.
Real work should not be handed to an AI agent before the manual process is well understood. A useful rule is that automation is not ready if the person cannot clearly explain why the manual process works. The hard part is deciding when a task has moved from something still being learned to something safe to outsource to an agent. People using agents in daily work need a practical way to choose which tasks are ready for automation and which still need a human doing them by hand for longer.
Reqable MCP Server lets AI tools use Reqable’s API testing features through MCP. It is built in Dart and talks to AI tools over stdio. It needs Dart SDK 3.0 or newer, then runs from the command line after installing dependencies with `dart pub get`. A specific Reqable API host and port can be passed at startup; the default host is 127.0.0.1, and the port comes from the local Reqable proxy setting when available or falls back to 9000. It exposes more than 100 tools covering HTTP testing, WebSocket testing, environments, collections, live capture, SSL proxying, breakpoints, request rewrites, network throttling, and access control. AI tools can turn captured traffic into cURL commands, open it as a new API tab, or add it to a collection.
RAG systems can give answers that sound right while the retrieved material does not truly support them. In live products, teams often do not have ground truth answers, and using GPT-4-level judges for every answer can be expensive. cgs-rag is an open-source Python package that checks whether a RAG answer is grounded in the provided context. It aims to work without ground truth answers or high-end judge models, and it runs in about under a second on a CPU. It combines token-confidence, NLI entailment, and cosine attribution into one risk score. It also separates honest uncertainty from confident false answers, so a model is not punished for saying it does not know when the evidence is missing. It works best on fluent answers that drift away from the evidence, and it is weaker on short answers about one entity. Different domains still need tuning with a small labeled sample, and it can be installed with pip install cgs-rag or tested through a reference app.
Skills Hub is a public GitHub repository with 67 ready-made skill sets for AI coding agents such as Claude Code, Codex, Cursor, Copilot, and OpenClaw. Each skill set contains focused instructions, workflows, and tool connections for a specific kind of task. Users can clone the repository and copy the skills into the right folder for their agent, such as `.claude/skills/` for Claude Code, `$CODEX_HOME/skills/` for Codex, or `.cursor/skills/` for Cursor. The skills are grouped into areas such as security, development, productivity, testing, writing, Chinese content work, Supabase, DevOps, and website generation. Examples include agent design patterns, API specification writing, code cleanup, unit test generation, email drafting, long document summaries, and security review workflows. The repository also points people to an MCP Registry for finding MCP servers. It uses the MIT license, has 27 GitHub stars and 1 fork, and has no published releases yet.
Token use can grow quickly in AI agent workflows. Tool calls, retries, planning steps, and long context windows are named as likely causes. The central question is which part uses the most tokens when agents run in production. This matters because reducing agent cost starts with finding the exact step where token use expands.
AI memory systems can look good on final answer quality while still pulling the wrong stored information. PrecisionMemBench measures retrieval directly by comparing returned items with fixed belief IDs. It calculates precision and recall from that overlap, without a generative judge, judge prompt, equivalence rules, or a scoring layer that a provider can tune for itself. The benchmark has 89 cases and is open-source. Providers do not submit their own scores; the harness runs tests through each provider wrapper and publishes the results. In the shown results, tenure passed all 43 active tests and scored 1.00 mean precision and 1.00 mean recall. Several other memory systems had high recall but low precision, meaning they often found many relevant items while also bringing back too much wrong or extra material. Retrieval speed and ingestion time also varied widely across systems.
Haystack is an open-source framework for building AI agents and RAG apps that can run in real products. It lets teams connect steps such as search, reasoning, memory, and tool use into one AI workflow. It works with tools and services including OpenAI, Anthropic, Mistral, Hugging Face, Weaviate, Pinecone, and Elasticsearch, so teams are not forced into one vendor’s stack. The same building blocks can be used for prototypes, testing, and production deployment. Its pipelines can be saved and reused, and the project highlights cloud-flexible deployment, Kubernetes support, logging, monitoring, and deployment guides. Main use cases include RAG, AI agents, multimodal AI, conversational AI, and content generation.
A local RAG evaluation was run over about 26 dense technical books. The books contain formulas, tables, exact numbers, and parameter values, so copying one figure incorrectly counts as a real failure. The setup used Strix Halo, 128GB of memory, Ollama, and fully offline models. The evaluation had two layers: retrieval and LLM-as-judge. Retrieval looked solved, with Recall@8 at 100% and MRR around 0.98. The uncertain part was the judge layer. The answer model was qwen3.5:122b, while the judge was llama3.3:70b-q8 from a different model family to reduce self-bias. Across 4 books and about 80 questions, the averages were about 91% correctness, 89% relevance, 60% faithfulness, and a 10% hallucination rate. The numbers are hard to reconcile because an answer should not be 91% correct if 40% of its claims are unsupported. The likely explanations are that answers add extra unsupported detail, or that the judge is too strict when it breaks answers into small claims. The judge has not yet been checked against hand-labeled answers, and the open question is whether measures like Cohen's kappa, reasoning judges, DeepSeek-R1-distill, or Llama 4 would give more trustworthy results.
The idea is to train one fixed memory model once and use it with different LLMs. This would only work well when the memory model and the chosen LLM share the same hidden state meaning. A separate small model would translate the hidden state between the two systems. That translator is meant to be only tens of megabytes, or available from a shared repository. The idea combines inspiration from the Titans memory paper with RescursiveMAS work on training small models to translate hidden states. The goal is better LLM memory without relying on vector databases, MCP tools, or elaborate prompts. The repository is open for others to test, break, disprove, or find errors in the current results.
AI agents can look strong in prepared demos but fail when they are put into messy real work. Many common examples are too simple, such as summarizing a document or answering questions about a spreadsheet, so they do not show how an agent would fit into a real company process. The workflow in question is sales operations. Its data is spread across a CRM, several internal tools, and manual handoff steps that are not well documented. An agent that could handle even part of this work would be useful, but there is doubt that the technology is ready beyond large, well-funded pilots. Useful evidence would include a real production deployment, the tool or platform used, the exact work the agent took over, and where it failed or disappointed.
A local Mac document app was tested by asking several small models questions about the same private file collection. The test used one unchanged set of 30 documents and the same 20 questions, with known answers used to check correctness. Apple Intelligence models were limited on medical and legal writing, so other models that can run under MLX were compared for real document answers. Qwen3 4B performed best, with 83.3% correct answers, 5.3 seconds warm latency, 6.3 seconds cold load time, and a 2.3GB size. Llama 3.2 3B reached 72.2% correct with a smaller 1.8GB size and faster 2.2-second warm latency. Phi-3.5 mini reached 66.7%, while Apple FM also reached 66.7%. Qwen3 8B was larger at 4.6GB but only reached 72.2% correct, with a much slower 19.5-second cold load time. The next round will test larger legal and medical document sets and the Qwen3.5 model family.
REAP is presented as a way to automatically build benchmarks for coding agents from real interactive production usage. The main idea is to turn the work a coding agent already sees in live use into evaluation material, instead of relying only on hand-made test tasks. The available information does not confirm performance numbers, cost savings, setup steps, or whether the code is open source.
A small media agency wants a central knowledge base that brings together client documents, research references, old emails, and other internal material. Staff should be able to ask questions, find past themes, and reuse earlier ideas when planning new work. The system should connect with Claude so that, when someone is working on a topic, the AI can bring up older findings that may help. New files added to Google Drive, local drives, or similar storage should also be added to this searchable knowledge base. The desired setup needs to be stronger than a personal notes tool, but still cheap, simple to deploy, and easy to maintain. One suggested path is a managed RAG service such as AWS Bedrock Managed Knowledge Base, which can handle file intake, search, and AI connection with less custom building.
Prompt caching can lower costs when the same opening part of a prompt is reused. Big LLM providers often mention this across pricing pages, documentation, or API notes, but the cost impact is not always easy to understand. Two prompts with nearly the same content can cost different amounts if their parts are arranged differently. Putting the user question, variables, timestamps, metadata, or other request-specific details near the start can break the stable prefix that the cache depends on. The practical rule is to put repeatable content first. System instructions, fixed rules, examples, schemas, and output format requirements should come before dynamic user input and request-specific data. For products that call LLMs many times, this small prompt design choice can meaningfully affect token costs and return on investment.
Testing AI responses only against a fixed dataset can miss problems that happen in the live product. A setup that scores prepared inputs during code review and blocks weak changes can catch many regressions before release. But real user requests can change over time, and a test set may not include the new ways people ask for help. In this case, output quality quietly got worse for one slice of real production traffic for weeks while every offline test still passed. The next step is online evaluation: sampling and scoring real production responses instead of only checking a fixed test dataset. The hard questions are whether to score all live traffic or only a percentage, when full scoring becomes too expensive, and how to alert on quality drops without creating noisy alerts.
Running a small language model directly inside a mobile app can still be too slow for normal use. Qwen 1.5B and Phi-4 mini gave poor results and repeated themselves too much. Qwen 3 4B is being tested instead, with an empty think tag forced in order to skip the reasoning path. The prompt cache is about 18,000 tokens, and the full prompt stays below 20,000 tokens. The cache is large because it carries many settings and conditions for the model. These choices were meant to make the model faster on the device, but each response still takes 2 to 3 minutes to generate. The target is to get replies under 1 minute after each user input.
The Supreme Court birthright citizenship ruling in Trump v. Barbara is a 194-page document. It includes a Roberts majority opinion, a Jackson concurrence, a Kavanaugh opinion that partly agrees and partly disagrees, and three separate dissents. The phrase “subject to the jurisdiction” changes meaning depending on which opinion it appears in. That makes the document difficult for simple search systems that split text into chunks and match similar wording. PageIndex handled it with a vectorless, tree-based approach. Its strongest result was finding the right opinion and section when comparing Kavanaugh’s reasoning with the majority’s reasoning. The answers also pointed to exact blocks and pages, so the PDF could be checked directly.
In a test of memory for agents that handle many conversation sessions, putting everything into one vector index did not clearly beat simpler search. The benchmark used LoCoMo, a dataset of real multi-session conversations with about 5,900 turns and 1,531 answerable questions. Six retrieval methods were compared: keeping only the most recent turns, BM25, two embedders, and BM25 mixed with each embedder using RRF. The recent-turns method scored about 0.024, which means it retrieved almost nothing useful, and it lost across all 10 conversations. The needed fact was often in an older session, outside the recent window. A single vector index with a strong embedder was only roughly tied with BM25. mxbai-embed-large scored 0.526, while BM25 scored 0.552, and the difference was not statistically clear. The practical result is that a cheap, self-hostable hybrid of BM25 plus an embedder may be a better default than using a vector database by itself.
AI agents can store false information in long-term memory when they save whatever the model decides is worth remembering. Once a made-up fact enters durable memory, it can be retrieved and repeated for weeks. Confidence scores are not enough because a model can sound very sure about a claim that is not in the source. A claim should not become durable memory based only on confidence. It first needs a source-grounding check that does not depend on the model’s own judgment. At least one cited evidence snippet must actually appear in the original source text, using a normalized text match and a token-overlap fallback for small punctuation or formatting differences. If the cited evidence is not found, the claim stays as a reviewable signal and is not saved automatically. High-impact claims, such as forecasts, commitments, or deal risks, still need independent confirmation or human review even when the cited text is found. The limit is that lexical grounding proves the quoted text exists, but it does not prove the model’s interpretation is correct.
OKF, short for Open Knowledge Format, is an open format for AI agents published by Google Cloud on June 12, 2026. The main idea is to stop explaining the same codebase or system to an AI agent every session. A project can include a `.okf/` folder with markdown files and YAML frontmatter that agents can read. Only one field, `type`, is required. It does not need an SDK, a schema registry, or lock-in to one vendor. Unlike a single CLAUDE.md or AGENTS.md file, OKF is meant to work more like a knowledge graph, where related ideas link to each other through normal markdown links. The files can live in git beside the code, so changes to project knowledge can be versioned like code. The claimed benefit is that many tools, including Claude Code, Cursor, Codex, and other agents, could read the same project memory. OKF and RAG solve different memory problems: OKF is for structured project knowledge people maintain, while RAG is for pulling useful information from larger document sets when needed.
LongMemEval is used to judge AI memory systems, but its results are being questioned. The benchmark came out in late 2024 and mainly tests whether a system can answer questions from a chat transcript. That covers only a small part of what a real memory system must do, such as keeping useful facts, finding them later, and avoiding wrong memories. Top scores are now clustered around 90% to 95%, so the test does not clearly separate stronger systems from weaker ones. A bigger concern is that high scores with little detail can get more attention than lower scores with a clear method. For example, a fully explained 81.5% result can be overlooked beside a 95% claim that does not show how it was run. A better setup would use a trusted group, a hidden test set, and the same fixed models for everyone.
Deterministic AI means the same question should produce the same answer. If the answer changes, there should be a clear reason, such as a changed setting or input, not unexplained randomness. Even with temperature set to 0, the same prompt can sometimes produce different answers. The practical issue is which methods can make output repeatable today, how reliable they are, and whether deterministic inference is used in production or mainly for evaluation and research.
RAG is a way to make an AI answer using relevant source material instead of relying only on what it learned during training. An LLM only knows the information it was trained on, so it can give a confident but wrong answer when asked about private documents, recent events, or internal data. That kind of wrong answer is called a hallucination. RAG reduces this by breaking documents into smaller chunks, turning those chunks into embeddings, and storing them in a vector database. When a question comes in, semantic search finds document chunks with a similar meaning, not just matching words. Those chunks are then passed to the LLM as context, so the final answer is based on the retrieved material. The key idea is that a question and a document can be matched by meaning inside vector space.
Teams running AI agents or multi-step LLM workflows in production can struggle to answer basic operating questions quickly. A workflow may suddenly cost 3 times more, but the team may not know which step caused the spike. A specific customer or workflow may be creating unusual retries, which can waste budget. Some workflows may consume money without making real progress. A failure may or may not affect the client outcome, and teams need to connect those events clearly. As AI workflows become more complex, normal logs, dashboards, and observability tools may not be enough to answer these questions fast.