Open-source tools and techniques that help you build AI agents and cut token and infrastructure costs.
Lemonade v10.8 is an update for running AI models on a local computer with less manual setup. The release had 20 contributors and was completed in 7 days. It can now unload idle models by itself and shrink the KV-cache to free graphics card memory while the system is running. Models that should stay ready can be pinned so they are not removed from memory. Lemonade also chooses the context length automatically based on available memory and the model design, instead of making the user tune it by hand. When a larger model is needed, work can be sent to OpenAI-compatible providers such as Fireworks, OpenRouter, Together, and OpenAI. This cloud offload sits beside local models as an option, while local running remains the default. LMX-Omni adds more image generation controls, such as image size and steps, and can pull or share custom Omni models through Hugging Face. Local models can also be called as MCP tools, which makes them easier for other apps or agents to connect to.
A $20,000 local AI rig is not free after purchase. Long AI inference runs keep using electricity, and that cost can materially change the math. The example assumes a serious machine with two high-end GPUs, enough system memory, and enough VRAM to run a useful model at a usable speed. It also assumes about $200 per month in extra power costs from regular heavy use. The comparison is a hosted AI subscription that costs $200 per month and has no upfront hardware cost. Under those assumptions, the local setup becomes cheaper only around month 27. Before then, the local rig is worse on pure cost, even if the upfront purchase makes the ongoing cost feel smaller.
On an RTX 5090, the Gemma 4 31B instruction model in GGUF format handled about 80,000 context tokens instead of about 35,000. A setting first used for DeepSeek Flash also worked with Gemma 4. The working setup uses `GGML_CUDA_NO_PINNED=1`, context size 80000, `flash attention` on, automatic fitting off, `backend sampling` on, and parallel runs set to 1. In the llama.cpp web interface, the `Backend sampling` checkbox should be turned on. This can help local AI agent experiments that need to read long material at once, but it depends on a high-end RTX 5090 and a specific runtime setup.
Very long AI conversations or work sessions can be handled by reading them in stages instead of forcing the whole record into one prompt. The method starts with a heavily compressed version so the model can build a broad outline. It then reads less compressed slices to add more detail. At the end, it can read small verbatim chunks when exact detail matters. The idea borrows from diffusion, where something starts blurry and becomes sharper, but it does not use the formal math behind diffusion models. Each slice is compressed so it fits inside the context window, meaning the model only needs the current slice, the user request, and the current draft output at one time. The model is also told which pass it is on, so it knows whether to make an outline or fill in details. The main goal is to preserve non-local information, meaning patterns or dependencies that only appear when the whole session is considered.
mcpsnoop is a transparent proxy placed between an AI client and an MCP server. It shows the real tool calls and replies passing between apps such as Claude Desktop, Cursor, or Claude Code and the server, live in a terminal. The official MCP Inspector connects as a separate client, so it can miss what the real client did or failed to do. mcpsnoop sits in the actual path and displays JSON-RPC requests, responses, notifications, and server error output. For a normal server, the server launch command is wrapped with mcpsnoop --; for a server reached through a web address, mcpsnoop can sit in front and forward traffic to the original server. It can replay one captured tool call against a fresh server copy, inspect the features agreed at startup, show full JSON frames, flag pending, slow, and failed calls, and filter the stream by tool, status, direction, method, ID, or text. Installation is available through Go, Homebrew, or prebuilt release files. Because it runs the server command it wraps, untrusted servers should be handled in an isolated environment.
Google Research's TabFM is a foundation model for data arranged in rows and columns. It can handle classification and regression when a table mixes number columns and category columns. It does not need fine-tuning or hyperparameter search. Example rows are given as context, and the model makes a prediction in one forward pass. This can reduce the work of training and testing a separate model for tabular prediction tasks.
Basemind is a local index for coding agents that helps them understand a code repository without reading whole files first. It runs on the user's own machine, connects to agents through MCP, and builds a code map across more than 300 programming languages. It also includes Git history and blame data, plus document search across more than 90 file formats. For many questions, it returns function signatures, line numbers, and structural clues instead of dumping full source files into the model. When the agent needs more detail, it can expand only the specific function body that matters. Basemind is written in Rust, released under the MIT license, and can run as an MCP server, a command-line tool, or a Claude Code plugin. The tradeoff is freshness and startup time: a large repository can take time to scan the first time, and the index can lag behind recent edits until watching or rescanning catches up. Nearby tools are moving in the same direction, including local model optimizers, dependency graph servers, richer code relationship graphs, and retrieval-first systems that prepare small evidence packets before asking a model to reason.
Long Claude Code sessions can waste tokens when replies and terminal logs stay in the conversation after they are only needed once. justsaydone focuses on making Claude Code replies up to 95% shorter to cut output token spend. Related tools point at the same problem from different angles: trim verbose agent replies, hide or delay raw terminal output, and avoid carrying large repeated logs through every later turn. Claude Code Eco reports 31% to 73% fewer output tokens in its normal saving mode while keeping critical findings, and up to 75% fewer tokens in a stronger mode. Boost uses a local CLI layer to filter noisy build, install, and search output before it bloats the context window. PxPipe takes a different route by turning long text context into image input, with claimed 60% to 70% token savings on Fable 5 in its own tests. The shared idea is simple: before changing the model, reduce how much the agent reads and says.
BaryGraph treats a relationship in a knowledge graph as its own searchable document, not just a line between two items. Each relationship is called a BaryEdge and has its own embedding vector. Pairs of BaryEdges can be combined into MetaBary triads, which are meant to reveal hidden links between ideas that are far apart in embedding space. The project targets a weakness in basic vector search: it often finds nearby text, but can miss two sources that describe the same underlying idea in very different language. One example is a flyby anomaly in orbital mechanics and an anomalous residual in stellar dynamics, which may be related in structure even if they do not cite each other or sit close together in search space. The system runs locally with MongoDB Community, mongot, and nomic-embed-text over the full English Wiktionary, covering 6.6 million documents. An MCP server is available, along with a preprint and benchmark CSV files.
These are notes from a conversation with the CIO of a large enterprise, discussing agentic systems, ontologies, and semantic layers. Large enterprises are focusing heavily on building internal agents rather than customer-facing ones, mainly to cut labor costs, but the infrastructure to support this is still immature. Enterprises sense they need better context management but don't yet have settled terminology for it. In practice, many point agents at fragmented internal systems and hope the model can infer business meaning across them on its own — an approach that breaks down quickly once it hits production. A core issue is that static ontologies (fixed models of business concepts) are outdated the moment they ship: the real world changes daily, but the ontology or semantic layer (the layer that maps raw data to business meaning) only gets updated once a quarter. Even human organizations get restructured every few years, underscoring how constantly reality shifts.
Multi-agent workflows can fail when each step decides by itself that its own work is good enough. If the first agent does research, marks it complete, and passes it on, the next agent may trust that mark and build on a mistake. By the time the final agent returns an answer, every gate may show success even though nobody independently checked the early work. This is self-certification: the system can look healthy in the logs while producing a wrong result. The check is to mark every point where the producer of a result also decides whether it moves forward. Prompt instructions that let an agent continue based on its own sense of completeness or confidence are warning signs. The fix is structural, not just a stronger prompt. The agent that produces the output should be separate from the role that judges it.
AtlasMind is an open-source VS Code extension in Beta. It is available through the VS Code marketplace and a public code repository. It can connect to more than 20 AI providers, including local models. It includes 15 dedicated agents, more than 40 custom skills, and support for MCP servers. A privacy dashboard lets people limit which models can receive personal or confidential data. A testing dashboard includes 23 testing protocols that agents can use automatically. A custom long-term memory system is built for SSOT and session-specific context. The ideation dashboard is still rough and is not the main focus for testing yet.
A mid-sized property management company runs about 340 units across four properties. It has too many missed calls and frustrated tenants, so the team is considering an AI voice receptionist for front-desk overflow. Simple calls, such as asking for office hours, look easy in sales demos. The concern is whether the system can handle harder calls, such as an upset tenant asking about a maintenance issue that has been open for three weeks, or a lease renewal with special negotiated terms. Those cases are not simple scripts. The practical question is what an AI receptionist cannot handle in real operations, especially where it has failed rather than where it looks good in demos.
A team compared three agent evaluation platforms: Testmu, Patronus, and Confident AI. Testmu covers the widest range of platforms, but it has the highest base price. Patronus is strongest at adversarial testing, but its overall scope is narrower. Confident AI is strongest for continuous checks using production traces, but it is weakest for multi-turn conversations. Each tool wins in a different area, and the team’s use case needs all three strengths. The practical choice is whether to buy one tool and accept gaps, buy two tools and connect them, or build custom work on top.
Godcoder is an open-source AI coding agent that runs on a personal computer. The user adds their own LLM key and connects directly to OpenAI, Anthropic, or any OpenAI-compatible model provider. Source code does not pass through a middle server; model requests go from the user’s machine to the chosen provider. It supports separate ask, plan, and coding modes. It can edit files in place, show code changes, roll work back to checkpoints, and continue past sessions. It also includes a terminal, file browser, and session history. An optional context engine can search large codebases using meaning-based search, code structure, call relationships, and keyword search. Prebuilt installers are not ready yet, so users currently need Rust, Node.js 20 or newer, and npm to build it from source.
Microsoft Research’s NextLat is a training method that tries to reduce the limits of predicting only the next token. A normal transformer learns to guess the next token, while NextLat also trains it to predict its own next latent state from the current latent state and the next token. This pushes the model to compress earlier information into a smaller belief state that can support reasoning and planning. Predicting in latent space may also give the model richer training signals than only choosing one token from a large list. The method can use recursive multi-step lookahead for self-speculative decoding, which is claimed to make inference up to 3.3 times faster. The item includes links to a blog, code, and paper.
A simple retrieval-based AI system can quickly turn into a large set of hardcoded decisions. As features grow, the application may need to choose the model, choose the retrieval method, handle memory for conversations, verify answers, route tool calls, use caching, and manage the budget. Putting all of that inside normal application logic can make the system rigid and hard to change. The proposed approach is to let the application describe the goal, not the full execution path. A separate runtime would decide which model to use, which retrieval method to use, what memory to include, whether verification is worth the cost, and whether a cheaper strategy can still do the job. The idea is compared to a database query planner, which figures out an efficient way to run a request internally. The approach has been implemented in Python and Go, with a whitepaper and code available.
Speculative decoding is a way to make large language models produce answers faster during inference. A small and fast draft model first guesses several upcoming tokens. A larger and slower target model then checks those guesses in parallel. This lets the system handle more than one token at a time instead of moving strictly one token after another. The goal is faster text generation without lowering the quality of the final answer. SGLang says it reached state-of-the-art latency for LLM inference serving by using Modal and Z.ai’s DFlash speculative decoding models. Papers with Code lists the method, related papers, and later papers that cite the original work.
Strata-K is a logic language where an AI model turns an English policy into rules, and a compiler plus a deterministic engine checks and runs those rules. For example, a policy like “do not clear a trade if any owner within two hops is sanctioned” can be translated into rules, then checked for mistakes such as using a name that was never defined. The checker returns stable error codes, the exact source location, and sometimes a fix that another program can apply automatically. After the rules pass checking, the engine runs them in a predictable way and can return the full reason why a result is true. The current open-source code is a CPU reference implementation in Rust. It supports stratified Datalog, negation, aggregates, stable models, and exact probabilistic queries. The planned feature that treats AI models as predicates inside the type system is designed but not built yet. The code uses MIT/Apache licensing.
An experimental scatter sampler has been added for llama.cpp. It slightly redistributes probability only among the top next-token candidates that were already selected. The goal is to make generated text less stiff without raising temperature and bringing in weak low-ranked choices. The name uses a light-spreading metaphor, but the method is not physics; it is a cheap smoothing step across nearby token ranks. The top-ranked candidate can give a little probability to the second, third, or fourth candidate, and a fifth-ranked candidate can exchange probability with nearby ranks. It is currently implemented and tested through the llama.cpp sampler API and can be used in a sampler chain under the name scatter.
HRM was used as the base of a document search model. The goal was to handle BRIGHT, a test where the right document often requires several steps of reasoning instead of simple word similarity. The model is small, with about 0.6 billion parameters, and it was trained on one RTX 3060 Ti with 8GB of memory. On the BRIGHT average nDCG@10 score across 12 areas, the original setup scored 18.1, query rewriting scored 34.3, and the merged setup scored 33.7. When the number of recurrent loops was reduced in the pony area, accuracy fell each time, which suggests the looped reasoning structure helped retrieval. The main weakness is knowledge. The base model was trained on a narrow text set for efficiency, so it struggles more in areas that need broad background knowledge. The weights are released under Apache-2.0, and the full BRIGHT evaluation harness is included in the code repo.
Teams using AI APIs can face large bills when basic controls are missing. A normal monthly AI bill of about $10,000 turned into more than $5,000 in one day after a bad loop kept calling the model. A production Gemini key was also used inside an AI coding tool and a command-line tool, with no separate development key and no clear spending limit. To the provider, the activity looked like valid use, but internally it was not where that key was meant to be used. Some code used API keys directly outside the normal key management system. When costs rose, the spending was visible, but it was hard to quickly find the source, the owner, or whether turning it off would break something important. The real issue is weak operating control around repeated calls, key separation, usage tracking, and ownership.
SwiReasoning is a technique that lets an AI model switch between 'explicit reasoning' (thinking out loud in text) and 'latent reasoning' (thinking internally without writing it out) depending on the situation. Doing this can sharply cut the number of tokens (units of text the model processes) needed to solve a problem, making responses feel faster overall. One person tested it on the Qwen 3.6 27b model and found the answers were more accurate and solved problems much faster. The raw tokens-per-second speed was actually a bit slower, but because far fewer tokens were needed in total, the overall experience felt quicker. The technique itself is about nine months old, with a GitHub repository (sdc17/SwiReasoning) and a llama.cpp implementation already available. Despite this, it hasn't caught on widely, which prompted the question of what the catch might be.
MiniMax M3 was run on a very large personal machine with 448GB of VRAM. The setup used two 96GB RTX Pro 6000 Max-Q cards, eight 24GB RTX 3090 cards, two 32GB RTX 5090 cards, three power supplies, 128GB of DDR5 memory, and a Threadripper 9960x processor. The model was reduced with AWQ-INT4 and served with vLLM, using PP and TP groups to split the work across many graphics cards. Speed was about 30 tokens per second for one single stream and about 960 tokens per second when processing a batch. The setup can provide a 1 million token context for one user, but the goal is to support four users at once. The best way to place and share that long context across users is still unresolved.
tokenscope is an open-source tool that shows where an AI coding agent spent its tokens during a work session. It reads local session logs and finds cases where the same files or content were sent again and again, or where very large tool outputs took over the context window. Each finding includes a suggested fix and an estimated saving. It can run without a full install through `npx tokenscope-ai`, or it can be installed globally. Claude Code logs under `~/.claude/projects/` are supported in version 0.1, while universal proxy mode, OpenAI raw JSONL logs, Gemini CLI, aider, and adapter support are listed as future or specification-stage work. Results can appear in the terminal or as a single HTML treemap report that opens in a browser and can be shared with a team. Token counts use a local tokenizer and are estimates, with about a 5% margin, so the goal is to spot waste patterns rather than match a bill exactly.
In local RAG, the main slowdown can be prefill, not decode. RAG adds thousands of tokens of retrieved material to each prompt, so the model must read a long context before it can start answering. On unified memory machines such as Strix Halo, decode can still look fine on some MoE models at about 25 to 40 tokens per second, but prefill can be much slower than on a discrete GPU. A single 24GB discrete GPU may process the same long context in a few seconds, while a unified memory setup can make the user wait 20 to 60 seconds before the first token appears. Unified memory can still be fine for batch work where waiting is less painful. For interactive RAG, a budget build should leave a free PCIe slot so a discrete GPU can be added later to handle prefill.
LLM app security testing is about more than servers, login, or access control. The practical concern is how the model behaves when people give unexpected instructions or move outside the intended flow. Many setups appear to depend on prompt testing and simple guardrails. Tools such as Garak and OWASP LLM are mentioned in the field, but it is unclear how often they are used in real projects. The useful comparison is how teams test model behavior, what tools they trust, and how systematically they handle unusual user actions.
This project tackles a specific fear about AI agents: the danger isn't the model saying something dumb, it's the model actually executing something destructive, like an accidental file deletion with no undo. The tool works as a proxy sitting between an app and any OpenAI-compatible LLM — changing one line (the base_url) routes every tool call through a policy check before it runs. Rules are written in YAML, for example blocking rm -rf commands or forcing dry_run (a preview mode that shows what would happen without actually doing it) on deployment actions. When a call gets blocked, the model is told why, and it usually reconsiders its approach instead of just failing. A CI evaluation reports a 90.9% attack catch rate (40 of 44 test attacks blocked) and a 5.0% false-positive rate (1 of 20 legitimate calls wrongly blocked). The 4 missed attacks are deliberately left in the test set: base64-encoded secrets and non-English prompt injection attempts (attacks that trick a model into unintended actions). The code is MIT-licensed and a demo runs without needing an API key.
Yaz from Tilelli Lab is an open-source local language model presented as a research prototype. Its main idea is editable knowledge: facts can be added, changed, or removed without training the model again. The intended use is custom knowledge that needs to stay current, handled more like a simple database than a fixed model. It is described as running locally on a computer’s CPU. It is also designed to answer “I don’t know” when it is unsure instead of making up an answer. Tilelli Lab also points to two earlier releases: Atome LM, which ran on a $5 ESP32 chip and included 12 AI apps, and Tilelli LLM, which ran on a CPU and also emphasized honest uncertainty.
A Qwen3.5-9B based fine-tune called Qwythos-9B was converted to GGUF format (Q4_K_M and Q8_0 quantization) and benchmarked on GSM8K, IFEval, and HumanEval using the lm_eval harness against a llama.cpp server, exposing several hidden pitfalls. Qwen3.5 models return their reasoning content in a separate response field from the main answer. If the llama-server isn't started with the --reasoning-preserve flag, the benchmark reads an empty response and every score drops by 50 to 80 percent with no error at all, just plausible-looking bad numbers. IFEval carries undocumented dependencies, langdetect and immutabledict, that aren't listed anywhere and surface as a ModuleNotFoundError partway through an hours-long run. HumanEval's built-in lm_eval task assumes a local-completions backend rather than chat completions, requiring a custom scorer that hits the chat API, strips out thinking blocks, extracts the code, and scores it with the code_eval metric. Loglikelihood-based tasks like HellaSwag and ARC turned out to be dead ends across three different approaches.