Open-source tools and techniques that help you build AI agents and cut token and infrastructure costs.
A local coding setup using Qwen 3.6 27B UD_4 did not work well for coding help. The machine used an RTX 5090, a 131K context, no KV quantization, and a model quantized by Unsloth. Cline inside VS Code was chosen because it seemed close to Claude Code. Fable created detailed, step-by-step plans for a basic Python app, written clearly enough for a junior engineer to follow, and those plans were placed in the repo. Qwen still produced code with many mistakes and generated terminal commands with basic syntax errors. It could not reliably follow the detailed plan, even when the task was mostly to move through the steps. The expected level was not Claude Opus or Claude Sonnet, but the setup still failed at basic coding-agent work through the tool harness and terminal calls.
RAG systems often give wrong answers because they receive the wrong information, not because the model itself is inventing everything. The model usually answers from the material it has been given. If that material is weak, missing, or poorly matched to the question, the answer will also be weak. The biggest gains can come from splitting documents by their real structure instead of fixed sizes, adding reranking after vector search, and building a test set from real user questions. Changing models may matter less than improving how the system finds the right information. Reasoning models do not repair bad input; they can make a wrong answer sound more convincing. GraphRAG can help with complex questions that span many connected documents, but simple document questions may be solved by better splitting and search before adding a more complicated system.
A new llama.cpp change makes NVFP4 models run faster on ARM CPUs. The key idea is to turn UE4M3 values into normal numbers by looking them up in a prepared table instead of doing slower repeated decoding work. This brings the ARM path closer to the x86 path, where a similar lookup table optimization already existed. In a CPU test with a Qwen3.5 4B NVFP4 model using 4 threads, processing a 512-token prompt rose from 1.89 tokens per second to 9.97 tokens per second. That is about a 5.3x speedup. The change also reuses the existing shared lookup table, so the ARM and x86 versions stay more consistent.
gemini-search-mcp is a free MCP server that provides web search through Google AI Mode with Gemini. It is presented as requiring no API key and having no usage limit. An AI agent could use it as a tool for finding information on the web. The main value is avoiding separate search API fees and API key setup.
AgentTransfer is an open-source tool for sending large files between AI agents, or between an agent and a person. Each agent can get an email address, a folder, and an inbox, then send files up to 5 GB through short-lived download links. A new agent starts with 400 MB of storage, and a verified human owner can unlock 20 GB and a permanent folder. Files are transferred outside the chat window, checked with a hash so the receiver can confirm the file was not changed, and actions are recorded with signed receipts. It includes an MCP server for agent tools such as Codex and Cursor, and it can run as one Go program on a personal machine or through self-hosting. The project is early and does not yet include encrypted storage, virus scanning, some mail security checks, resumable uploads, or webhooks.
OneDev 16 Community Edition adds autonomous AI users to its self-hosted all-in-one Git service. An AI user can take an issue, use the issue details as the work request, open a controlled workspace, change code, and create a pull request. The same AI user can then answer review feedback, repair failed CI/CD builds, resolve merge conflicts, and keep updating the pull request until it is ready. AI users can also join code review as reviewers or assignees. They can inspect changes, leave line comments, approve changes, ask for revisions, and optionally merge after required reviews and CI/CD checks pass. The main design keeps AI work inside normal software team records: requirements stay in issues, execution happens in isolated workspaces, code changes move through pull requests, and validation stays in CI/CD. Rules can route matching issues to an AI user or require AI review through pull request settings and branch protection rules.
The same vector index can give a correct answer in one product surface and make things up in another when the query is built differently. A reliable RAG flow first turns the chat history and the latest message into one clear question that can stand alone. For harder questions, it can create 3 to 5 query variants or use HyDE, search with each one, and combine the results, but this should be limited because it adds cost. Search should combine meaning-based retrieval with BM25 and pull a wide set of about 50 to 150 results from each. BM25 helps catch exact names, product names, and IDs that meaning-based search can miss. The results are then merged by rank, narrowed with cross-encoder rerank to about 5 to 20 candidates, and filtered with a score threshold. The final answer should use only the supplied context, and say it does not know when the context is not enough. During ingestion, quality can improve by adding a short LLM-written note to each chunk, indexing that note in both search systems, adding likely questions or paraphrases for each chunk, splitting documents by meaning, and attaching metadata.
Most RAG systems fetch needed knowledge, use it for an answer, and then drop it. That is simple and often sensible. For long-running agents or tools, some repeatedly useful knowledge might be worth keeping as permanent memory. Saving everything would create confusion, so the system would need a way to decide what is important enough to remember. The main risk is that permanent memory may store stale knowledge or bad assumptions, which can make later answers worse.
tftf is an open-source tool for working with AI models that are too large to load into normal memory or VRAM at once. It handles the model in smaller tensor-by-tensor pieces instead of opening the whole model in memory. This can help with jobs such as LoRA merging and converting a model from one file format to another. The main audience is people who fine-tune large models. The tool is experimental, so important work should be checked carefully before relying on the output.
AI exercises for communication and sales training can work well as a fixed workflow when the evaluation method is clear. The task can be split into small steps, and each step has a known job. That can make the whole run cheaper, faster, and easier to control. An orchestrator approach looks more flexible, but it may be more than the job needs when the process is already predictable. The practical choice is either to start with an orchestrator and restrict it, or start with a fixed workflow and add orchestration only to the messy parts. Assemblix is presented as a self-hosted tool for helping non-developers build business dialogue AI agents.
Open-connector is an open-source tool that handles the login and authentication process AI agents need when accessing SaaS services like Gmail, Slack, or Notion. It supports connections to more than 1000 SaaS providers, and developers can plug it into their agent through whichever method fits their setup: SDK, CLI, MCP (a standard way for agents to use outside tools), HTTP, or OpenAPI. In practice, this means an agent builder doesn't have to write separate login logic for every single service — one gateway manages access across many of them.
A developer is building a fully local smart-home assistant using a Qwen 4B model (GGUF quantized format), Outlines to generate structured JSON, and FastAPI, MQTT, and PostgreSQL for the rest of the stack. The current pipeline splits a user's sentence in Python, then an intent classifier sorts it into one of five categories — Action, Preference, Status, Scene, or Delete — before a dedicated parser turns it into structured JSON sent to MQTT or the database. This works fine for simple, single-intent commands like "turn on the hall lights" or "what's the AC status." It breaks down when a sentence contains multiple intents at once, such as "turn on lights in the R&D room when I enter and save this as a preference, and also turn on 3 lights in the Software room" — which mixes an immediate action with an automation rule. Because the classifier assumes one command equals one intent, the small 4B model starts mixing up attributes between tasks or routing the whole request to the wrong parser as commands get more complex. The developer is now considering replacing the classifier with a small "planner" that breaks a sentence into a list of separate tasks instead.
Standard retrieval-augmented generation (RAG) works by converting documents into vectors and finding the nearest match by distance. The problem: compressing text into vectors throws away a lot of context, so nearest-vector search often misses evidence a model would recognize if it could actually read the surrounding material. Once recall starts failing, people pile on fixes — tuning chunk size and overlap, combining keyword and semantic search, adding rerankers, metadata filters, query rewriting, summaries, thresholds — but none of that changes the fact that vector search still isn't the same as reading the evidence. Attemory takes a different approach: instead of embedding chunks and searching by vector distance, it indexes the raw text, documents, or codebase into reusable KV (key-value) state. At search time, a local Qwen3.5 retrieval model attends directly over that indexed memory and the query, then returns compact evidence — memory IDs, snippets, or file and line ranges — rather than just matching compressed vectors. The developer reports state-of-the-art results on long-memory and code retrieval benchmarks with this approach.
Russian tech company Sberbank released a new large language model called GigaChat3.5-432B-A28B. It has 432 billion total parameters, but uses a mixture-of-experts (MoE) architecture so only 28 billion parameters are actually active when generating a response. A base, pretrained-only version was released alongside it. Notably, a GGUF version was made available on day one — GGUF is a compact file format that lets a model run on llama.cpp, a program for running models on personal computers. However, this GGUF support has not yet been merged into llama.cpp's main branch; it currently requires building from a separate pull request (PR).
Knowledge graphs are expected to help retrieval augmented generation (RAG) connect related facts that use different words. For example, one law may ban stealing, while another may allow taking food in a true emergency. Lexical search can miss that link because the words do not match. Semantic search may find the connection, but the needed text chunk can still fall outside the top-k window and never reach the answer. Knowledge graphs sound like a stronger way to store and update relationships, but they do not always perform better than other RAG methods. The weak point may be that the LLM fails to extract enough complex relationships during indexing, or that the embedding model cannot bridge the wording gap when storing and searching information.
A RAG test used about 17,000 support-ticket chunks in several languages, including German, English, French, and Italian. Each of 50 questions was evaluated against the top 10 retrieved results. The dense retrieval baseline with F2LLM-0.6B-Preview was already strong, with an nDCG@10 score of 0.8421. Adding three pointwise rerankers changed the scores to 0.8292 for cross-encoder/mmarco-mMiniLMv2-L12-H384-v1, 0.8360 for BAAI/bge-reranker-v2-m3, and 0.8427 for Qwen/Qwen3-Reranker-0.6B. MRR for results rated at least 2 on a 0-to-3 relevance scale fell from 0.980 in the baseline to 0.937-0.957 after reranking. The useful document was already almost always in first place, so reranking sometimes moved it lower. Open questions remain around whether other reranking models or pairwise reranking would help more than pointwise reranking.
The current setup runs a small Jina embedding model and a Jina reranking model in Q8 format inside Open WebUI. The next plan is to stop running the large language model locally and use the DeepSeek API instead. That would free local computer resources for larger embedding and reranking models. The main choice is whether to keep a small 0.6B model or move to a larger 4B quantized model. The intended use is writing and math, so accuracy matters more than simply keeping the setup light.
The European Commission selected the EUROPA consortium as the winner of the Frontier AI Grand Challenge. EUROPA is led by the Italian company Domyn. The project will build an open-source artificial intelligence model that supports all 24 official EU languages. The proposed model is planned to have more than 400 billion parameters, placing it in the same size range as very large frontier models. The Commission chose EUROPA to strengthen Europe’s ability to build advanced AI on its own infrastructure. The finished model is meant to be openly available, so businesses, researchers, and public institutions across Europe can use advanced AI across many languages.
AI agent token costs may not be mainly solved by switching to a cheaper model. In the examples collected here, bigger savings came from limiting what the agent is allowed to put into context. Large, unpredictable terminal outputs can consume many tokens quickly; one Codex example cut token use by about half by adding one AGENTS.md rule that capped shell output when the size was unknown. Tool definitions were another major cost. One team had 508 MCP tools and was spending about $377 per run because those tool definitions were sent again on every call; by not sending every tool description upfront, the cost dropped to about $29 per run. Another measurement found that about 67,000 tokens were already used before the person asked the first question. Choosing smaller or cheaper models can help, but the larger savings usually come from controlling tool output, tool lists, and the starting context.
A proposed llama.cpp change turns on -ffast-math for HIP builds. The benchmark used a gfx1151 AMD chip, RDNA3.5/Strix Halo with 40 compute units, and compared the change with the latest main code. On Qwen3.5-27B Q4_K_M, prompt processing rose by about 4.0% to 7.0%, depending on prompt length. On Qwen3-0.6B BF16, the gain was around 3% for shorter prompts and only 0.6% at 32,768 tokens. With FA=0, the two builds were almost the same. The practical result is a small speed gain for reading prompts on certain AMD GPU paths, not a broad change for every setup.
A coding agent can stop in the middle of writing code when it reaches a token limit. Building software usually needs many rounds of changes, not one perfect answer. When a large language model does not understand enough context, the person has to explain more and retry more often. The current setup can make builders think about usage and cost while they are still trying to shape the software. A cheaper way to run coding agents on personal computers, using affordable RAM and GPU hardware, would reduce that pressure.
Athena is a voice-to-voice AI assistant that runs on local hardware instead of sending speech to cloud services. It combines Whisper-small.en for speech recognition, Qwen3.5-397B for generating replies, Orpheus 3B for spoken output, and SNAC for audio handling. The runtime is written in C++ and uses a four-part process, with no Python needed during normal use. One optional Python script is used only once during setup to convert an emotion model offline. Athena can speak with emotional sounds such as laughs, sighs, and gasps. It can read basic emotion in a person’s voice, keep long-term memory and personality across conversations, and handle long conversation history. If the user talks over it, it stops speaking and keeps what it already said in the conversation context. It is designed to run without cloud services, telemetry, or API keys on a consumer GPU plus system RAM.
A traditional RAG app built on LangChain docs ran into trouble during data preparation. Chunking failed, and manually setting overlapping chunks took about 49 minutes to ingest into a vector database. The app was completed, but the retrieval step returned answers that were not relevant enough. The likely problem was a mistake in how the source material was ingested and stored. GraphDB and GraphRAG are being considered as alternatives, with the main question being when GraphRAG is useful, when it is unnecessary, and what tradeoffs it brings. There is also a cost concern if the graph keeps growing during each chat.
Excalibur is an open-source tool that lets product engineers hand coding tasks to an AI coding agent and manage the whole flow from planning to review. The Core version is released under the Apache-2.0 license and can be installed with npm, then run inside a user’s own code repository. Before writing code, it checks scope, evidence, and risk so the work can be built, validated first, or dropped. Users can choose how much control to give it, from answering questions and suggesting small patches to building a change in a separate branch or running a more automatic workflow. Larger jobs can be split across several AI agents, with each part worked on separately and checked by tests and review before merging. For cost control, Excalibur keeps a full run history, lets users restart from an earlier step, and aims to reuse unchanged earlier work from cache instead of running everything again. It supports OpenAI-compatible services, Anthropic, Ollama, and other providers, and can use stronger models for hard parts while using cheaper models for lighter text work. A local dashboard shows tasks, live progress, cost information, and run history, while writes, commands, and pushes require approval; it also includes sandboxing, secret blocking, and local audit logs.
kv-psi is a small reference implementation that uses Linux Pressure Stall Information to shrink an LLM KV cache when a computer is under memory pressure. A KV cache stores past context so the model can keep generating text without recalculating everything, and it grows during long chats or long document work. The tool reads memory pressure from cgroup memory.pressure or /proc/pressure/memory, then prunes part of the cache when pressure rises. It needs Linux with PSI enabled, Python 3.10 or newer, llama.cpp runner build dependencies, and a GGUF model. The basic flow is to run a simulator, build the llama.cpp runner, and download a small benchmark model if needed. In a recent Jetson result with 1536 decoded tokens, the fixed setup did no pruning and ended with a final KV size of 1547, while the PSI setup pruned 2 to 4 times and ended at 1004 or 1291. Speed varied by run order and pressure state: PSI showed 88.80 or 96.16 tokens per second, while the fixed setup showed 94.00 or 89.76 tokens per second. The benchmark runs both variant orders because cache state, zram/swap, and PSI averages can carry over from one pressure run to the next.
In multi-agent LLM workflows, the cost question is not only which model to use. The bigger issue is how to spend tokens and expensive reasoning carefully. The main goal is to save the strongest reasoning budget for decisions that truly need it. Broad research and mechanical work should go to the cheapest capable model, often in parallel, with only narrow and structured results passed upward. Final judgment should come together in one place instead of being scattered across many smaller calls. The workflow should summarize once and reason once, rather than repeatedly compressing and rethinking the same material. Retrieval should happen step by step, so the system brings in only the information it needs instead of loading everything at the start. When something is unclear, the system should raise the uncertainty to a stronger decision step instead of letting a cheaper model guess. These are general orchestration principles, not a fixed ranking of models.
Awesome Agentic DevOps is a curated list of AI agent tools and MCP servers for DevOps, cloud, SRE, and platform engineering work. It compares tools that can be used with Claude Code, Gemini ADK, OpenAI Agents SDK, and MCP. The list goes beyond discovery. It checks whether a tool is useful near real production work, whether it can access live systems, whether human approval is built in, whether logs or evidence are preserved, and how mature the tool looks. It groups official resources by practical use case, including AWS, Microsoft Azure, Google Cloud, GitHub, GitLab, Atlassian, Terraform, Pulumi, Grafana, Datadog, PagerDuty, SonarQube, Okta, Snyk, Wiz, Jenkins, and Argo CD. The safety guidance is direct: start in read-only or proposal mode, require human approval before changes, keep secrets out of model context, use least-privilege credentials, run dry checks, review Terraform plans before applying, and use CI checks before merging.
Security testing for AI applications and AI agents needs strong attack examples. Many red-team setups use a large language model to create difficult or unsafe inputs on purpose. The needed recommendations cover both closed-source models and open-source models that can produce realistic, challenging attacks. The attack types include toxicity, prompt injection, SQL injection, jailbreaks, indirect prompt injection, prompt leakage, tool misuse, multi-turn attacks, and other agent-specific attacks. A public benchmark dataset is also needed, so teams do not have to generate every attack from scratch. The ideal option is a golden dataset with predefined, high-quality attacks. Practical recommendations could include models, datasets, papers, or GitHub repositories used by people working on AI security and red teaming.
An agent that buys Solana tokens can mistake a coordinated scam token for a real one if it only looks at price and trading volume. This tool is offered as an MCP server and also as plain REST, with one main check that takes a token address and inspects wallet relationships. It traces whether top holders were funded from the same source, then looks for wallets that bought in the same block as a sign of coordinated entry. It also flags cases where several holders sell a large share in the same block, which can point to a coordinated exit. It checks the token creator’s past launches and labels the history as first launch, normal, poor track record, or serial rugger. It also looks for honeypot risks such as live freeze authority.
cs4ai is a Roslyn-backed command-line tool that helps AI agents work on C# projects by using code meaning instead of plain text search. When Claude looks for something like AddCQRS with grep, it can get many noisy matches from comments, strings, project file names, and lookalike names, so the agent reads too much and may choose the wrong place to edit. cs4ai starts a session from a C# solution, does a full build, and returns a session token. It can inspect a whole type, return a staleness token, update a method body, rename a symbol across all call sites, and verify the result with a real dotnet build and tests. Edits must include the latest staleness token, so an agent using an old view of the code gets refused and receives the current source needed to recover. Exit codes only say whether the command was valid; build and test results are placed in the response body and marked as new or preexisting. Changes are written directly to disk, and undo is left to git.