Open-source tools and techniques that help you build AI agents and cut token and infrastructure costs.
Model Shifter is a macOS tool that lets Claude Code users change models and effort levels with a manual car gear-style interface. Its dashboard shows token use in the last minute, remaining capacity in the current 5-hour usage window, and lifetime token use from saved history. Using the clutch while changing gears opens a new session to reduce cache burn. A later update connects the app to a physical racing simulator shifter, so moving the gear stick can switch Claude Code models. FANATEC hardware works by default, and Logitech mapping was also mentioned. The wider discussion is tied to Fable usage limits, a move toward usage credits, and attempts to combine Fable and Opus workflows to keep quality high while spending fewer tokens.
This workflow reduces two common Claude Code problems: making changes without enough confirmation and asking for approval for too many small actions. The main step is to add execution rules to the project’s CLAUDE.md file, so Claude Code knows which actions must be confirmed before it runs them. This helps limit scope creep, where the AI moves beyond the user’s intended task, and approval fatigue, where repeated prompts slow the work down. The workflow is marked as intermediate level and is tied to quality control, token saving, context management, debugging, and shipping. Its listed scores are 85 for usefulness, 70 for freshness, and 0.95 for confidence, with an active status.
A firsthand test measured speculative decoding by actual power use, not just speed, and the result was worse than expected. It did not make token generation faster, and it used more energy for each token. The setup used an RX 6650 XT graphics card with 8 GB of memory, Windows 11, llama.cpp with Vulkan, and llama-server with 8 slots plus continuous batching. The main model was Qwen2.5-3B-Instruct in a quantized version, and the smaller draft model was Qwen2.5-0.5B-Instruct in a quantized version. Both models ran fully on the GPU. The same 8 prompts were tested each time, each request generated 256 tokens, prompt cache was turned off, and both 1-stream and 8-stream cases were measured. GPU power was sampled with LibreHardwareMonitor about 6 times per second, then converted into energy used during each generation window. Idle power was about 19 watts, and the energy penalty became worse as GPU load increased.
In a firsthand experiment, GPT 5.5 was given an empty GitHub repository and set to work on it once every hour. The intended loop is to review past work, choose the next task, write code, run tests, and commit the changes. Its first commit did not build a working feature. It created a roadmap, a changelog, a state file, and a file recording its decisions. The AI later described the goal as “Autonomous Forge,” a safe AI maintenance manager for GitHub projects. The repository is public, so its decisions, test failures, fixes, and possible over-planning can be observed. It is still unclear whether this will become a useful tool.
An AI assistant summarized a document that belonged to another customer. This was not an outside break-in; the vector search returned the most similar text chunks it was asked to find. Vector search can judge similarity, but it does not know who is allowed to see each document. The real problem was the permission filter design. Rules for workspace, owner, visibility, role, and archived status were written by hand inside app code, with slightly different copies across three features. This filter ran on every retrieval and acted as the security boundary, but it had almost no testing. When one condition was missed, nothing crashed; the AI produced a smooth, confident answer using data the person should not have seen. Later review was also hard because logs showed what the app requested, not the exact permission rules enforced for each query.
A new quota monitor helps people who use several AI services, including Claude, Codex, Copilot, OpenCode, Z.AI, Kiro, and Antigravity. It shows usage and remaining limits from the terminal or Waybar. The goal is to make it easier to avoid suddenly running into a service limit during daily work. Feedback is being requested from people who switch between multiple providers, especially on which metrics would actually help them avoid blocked work.
A large language model pipeline is taking in many recorded meetings and voice calls. Whisper turns the speech into text, but the result is a huge, unstructured block of words. The transcript includes filler sounds such as “um” and “uh,” along with empty pauses. When that text is split and stored in a vector database, the RAG agent spends many context tokens on low-value text. Without speaker labels or clear document structure, the agent can lose the thread of the conversation or hallucinate. The practical issue is whether audio transcripts should be cleaned and structured before they are used, instead of paying OpenAI or Anthropic to process messy input.
When a model is fine-tuned on new data, poisoned data can secretly add hidden behavior that appears only after a specific phrase or pattern. The proposed method limits how the model is allowed to change, so it can only move within the range expressed by trusted LoRA adapters. Useful customization can still happen, but some harmful update directions become unreachable by design. In a company setting, a model may be fine-tuned on user data, outside data, or generated data, and even a small amount of poisoned data could create a backdoor behavior. This approach does not try to detect every bad data point; it narrows the space of updates the model is allowed to learn. The same idea could apply to a local or on-device assistant that keeps adapting to its user, letting it change only within behavior patterns already covered by a trusted pool of adapters. The approach was tested on 196 public LoRA adapters, including adaptive attacks built to bypass the defense.
An app needs an LLM adjusted for its own specific use case. The plan was to use a service such as Together AI or Fireworks to run a LoRA fine-tuned model and pay only for the tokens used. Those services no longer appear to offer the same pay-per-token pricing for LoRA fine-tuned models that they once did. Clear online information about the change is hard to find. Usage will be sporadic over the next few months while the app is tested, so renting a GPU full time is too expensive. The main need is the cheapest way to host a model with a LoRA adapter.
Paid voice services such as ElevenLabs can be expensive. Open-source text-to-speech models are available, but they are often hard to find and try because they sit inside GitHub repositories and require manual downloads. This tool provides a standalone interface for browsing text-to-speech models and testing their voices directly.
A multi-agent workflow using Gemini models takes about 2 minutes to answer even a simple question. The flow runs through input guardrails, a planner agent, several sub-agents, a response builder, and output guardrails. NeMo is used for the input and output guardrails. The system works, but every layer runs before the final response, so the full wait time becomes long. The main production questions are how to reduce end-to-end response time, whether guardrails should run one after another or in parallel, and how to stream answers while agents are still working. The example architecture is shared as a Smart Bill Book workflow agent diagram on GitHub.
A platform needs a large language model to refer to many existing nodes. When creating a DAG, the model must know many nodes that were already defined and point to them correctly while building the graph. The main challenge is giving the model enough context while keeping response time, cost, and reasoning quality under control. Context caching may help if most of the context stays the same across requests. Another option is Retrieval-Augmented Generation with a vector database. The concern is that this may only fetch a small set of similar items, while the model may need broad access to many nodes, not just a few close matches.
Rules written only inside a prompt are not reliable controls for an AI agent. The model may ignore them, earlier context can lose influence as the session gets longer, and the model may behave as if the rule no longer exists. Real rules should be enforced by the application around the model. The application should decide which tools the agent may run, which data it may read, and which actions it may take. Without that outside control, a rule in the prompt is closer to a request than a hard limit.
Mozilla CTO Raffi Krikorian will hold a live AMA on July 14, 2026, at 1 p.m. Eastern Time. Mozilla plans to publish its first State of Open Source AI report on the same day. The discussion will focus on how open source AI is actually used in production by developers, companies, and the wider AI ecosystem. Key topics include the hidden cost of “free” models, the real business cost of relying on closed tools, where companies get stuck when adopting AI, and how free capable Chinese models are changing market power. The report also draws on input from more than 950 developers about which AI tools they trust and why. A major theme is the agentic harness: the layer above the model that helps run AI agents, where Mozilla sees the next important fight for open AI.
AksharaMD is a local Python command-line tool for checking documents before they are added to an LLM or RAG system. It turns a file such as a PDF into Markdown, structured JSON, validation warnings, a run manifest, and chunk JSON. The manifest includes a 0 to 100 readiness score and a quality label such as HIGH, OK, RISKY, or POOR. It warns about files that need OCR, have too little readable text, contain broken character artifacts, or create unnecessary token growth. Optional add-ons cover OCR, image and table extraction, math OCR, audio transcription, and S3 input. The code is source-available under PolyForm Noncommercial 1.0.0, so commercial use may be limited.
Lians is an open-source memory engine for AI agents. It is aimed at agents that handle time-sensitive data or regulated data. Each fact is stored with two time views: when the fact was true in the real world and when the system learned it. This lets an agent recall only what was known as of a chosen past date. The full record is protected with an append-only hash chain, so hidden changes are easier to detect. Separate encryption keys for each subject make deletion provable while keeping the audit trail intact. Lians can run inside an app, on a self-hosted server, or in the cloud. It supports Python, TypeScript, Go, Java, and C SDKs, native MCP support, local SQLite storage, and self-hosted Postgres.
An independent researcher, working alone and self-funded with a single RTX 3090 GPU, released a preprint proposing a new memory design called a fast-weight memory bank. In this setup, the model writes values into a small bank of vector slots during its own forward pass (a single pass of input through the network to produce output), and later reads those values back not as ordinary data but as if they were model weights. Each slot is expanded by a hypernetwork (a network that generates the weights for another network) into a low-rank MLP layer that gets applied directly to the incoming token stream. The goal is continual learning during inference with no backward pass at all — no test-time training, no separate optimizer step, no cloning of weights, and no growing context window. The experiment used a tiny 3.08-million-parameter DeepSeek-style transformer fitted with an 8-slot memory bank, tested on a keyed multi-turn rule task: each conversation binds two key tokens to a brand-new rule, shows that rule exactly once as a 13-token sequence, then later asks about symbols the model has never seen paired with that rule. Because each turn is a fully separate forward pass, a rule can only carry over between turns by passing through the memory bank. Random guessing would succeed only 0.008 of the time, and removing the memory bank entirely (as a control) dropped performance to exactly that chance level everywhere. The main finding is that the approach works and generalizes: a single 13-token presentation of a never-trained rule let the model answer unseen queries correctly 0.79 to 1.00 of the time, replicated across two random seeds. The learned rule also survived even when its storage slot was physically evicted.
A mid-size logistics company is trying to use AI agents to help with freight routing. The agents would help choose carriers, combine loads, and make other routing decisions. A wrong answer can create real financial loss, not just a bad chat response. If a truck is sent to the wrong dock, the mistake may only appear in the operations dashboard after the damage is already done. The needed tool is not basic chat monitoring, but decision-level tracing that shows what inputs the agent saw and how it reached a choice. The system also needs a pre-execution check against business rules, such as a carrier's maximum capacity, before anything is committed to the TMS. Generic LLM monitoring tools seem built for chat use, not for blocking and checking very expensive dispatch decisions before they happen.
Speculative decoding has a formal guarantee: with rejection sampling, the final output distribution stays the same as the larger target model. The quality of the smaller draft model does not decide correctness. A weak draft model gets rejected more often, so it can reduce the speed benefit, but it does not change what the target model would say. This makes it different from quantization, where speed or size gains can come with a quality tradeoff. The acceptance rate is directly tied to the gap between the draft model and target model distributions: it equals 1 minus the total variation distance. The practical question is why a lossless speed method is not the default in more local setups, and what real speed gains different draft-target pairs actually deliver.
koder is a browser-based tool for coding and computer-use agents. It was built over 11 weeks with about 1,300 commits and is designed mainly for local or offline use on Linux. It is OpenAI compatible, so it can connect to local models or cloud models. The main tested setup is Linux with llama.cpp and Qwen 3.6 27b Q8, where it is reported to be stable. Results may change with other models, and Gemma 4 did not work well in this setup. The tool has been used for coding, reverse engineering with Ghidra, building tools, writing OpenSCAD with visual feedback, and online research.
MIRA is a multiplayer world model built by General Intuition, Kyutai, and Epic Games. It predicts and continues game-like action for four players in a Rocket League-style setting. It was trained on 10,000 hours of synthetic data. The model has 5 billion parameters. It can run a four-player scene at 20 frames per second on one B200 graphics processor. A playable online demo, a detailed technical report, a 1,000-hour four-player gameplay dataset, and a GitHub repository are available. An in-person demo at ICML lets people try it with PlayStation controllers.
GLM-5.2, a 744-billion-parameter open AI model, was run on a laptop with 12 CPU cores and 25 GB of RAM. The model uses a MoE design, so only selected parts need to be loaded for each request instead of the whole model at once. A small pure-C engine called colibrì kept the dense parts in memory, using about 10 GB, and loaded the needed expert parts from disk when required. The work involved quantization, streaming, MTP tuning, and help from coding agents. The result was real chat output, including Italian conversation, on modest hardware. The speed is still extremely slow at about 0.05 to 0.1 tokens per second on a cold run. The project is early and built by one person, with hopes that people using faster NVMe storage or more RAM can test it and share results.
In a RAG app, a vector database can accept a saved item before that item is ready for search. A user may upload a PDF, see a ready message, and then ask a question that cannot find anything from the new file. A permission change can also be accepted while old results remain reachable for a short time. This makes simple write latency a weak measure for real app behavior. A better test checks whether the database request was accepted, whether the new vector can be searched, whether metadata filters work correctly, whether recall has settled, and how much the full path costs in the cloud. VDBBench was used to measure these stages separately: accepted, searchable, fully indexed, and cost.
Anthropic published research examining what happens inside an AI model before it decides what to write. The research points to something they call a global workspace — a specific area inside the model where information gets gathered and processed before a response is produced. Anthropic released the code for a tool called the J-Space lens that lets people inspect this internal structure, and a partner built a demonstration applying it to Qwen 3.6 27B, an open-source model.
A smart home assistant built with a small 4B language model has trouble when one sentence contains several actions. “Turn on the bedroom light, make it warm white, and set brightness to 20%” looks like three actions, but it is really one desired final device state. If the system splits it into steps, the light may turn on with a default setting, then change color, then dim, which creates a poor experience. “Turn on the bedroom light and save this as my evening preference” should be split because it crosses two areas: device control and preference storage. Outlines can extract on/off state, brightness, and light temperature from one simple command. The harder case is a mixed command such as turning on three hall lights at 30% while turning off four bedroom lights, where naive splitting can damage the meaning of the sentence. The practical question is whether to classify the domain first and let the device agent handle all device attributes together, or use another planning method that works better with small language models.
Five real AI knowledge bases will be selected for free diagnosis and repair. The target systems technically run, but they forget information, ignore set rules, give conflicting answers, or waste too much of the context window. Eligible projects include RAG systems, project knowledge bases built on ChatGPT or Claude, vibe coding documentation and project rules, internal AI assistants, document collections prepared for an AI workflow, and web-based AI apps or chatbots. The main problems are cases where AI keeps repeating a mistake after being told the right information, or gives wrong answers even after all the documents were provided. Project details and problems can be sent by private message or comment. Confidential company data and sensitive information should not be shared.
Running Qwen 3.6 27B with MTP produced about twice as many tokens per second. That means the model generated text much faster in the same amount of time. The result led to interest in finding other MTP-capable models, including abliterated MTP models.
Local AI models can slow down sharply when a conversation becomes long. Even on an M5 MacBook Pro with 128GB of unified memory, a context around 16,000 tokens became a serious bottleneck in this experience. The main issue is the growing context, not just the model choice. A more workable pattern is to split a job into tiny parts, run each part in a fresh short session, and pass only the short result or summary to the next step. For an overnight scraping job that becomes a morning dashboard, many small workers can each extract one piece of information, then one aggregator can combine only the short summaries. This map-reduce style fits local agents better than one agent carrying a long history. The open question is which open-source agent orchestration frameworks support this stateless worker pattern well, since tools like CrewAI, AutoGen, and default LangChain often carry too much history forward.
Local LLM return on investment is often judged by decoding speed, which is how fast the model writes an answer. For agent work that reads a lot of material, prefill speed can also change the economics because it measures how fast the model processes input before writing. In one example, GLM 5.2 running on four NVIDIA DGX Spark machines with 4-bit compression, speculative decoding, and other tuning reached about 60 output tokens per second with 6 concurrent users. If that ran as a 24/7 agent workload, it would produce about 5.18 million output tokens per day, or about $22 per day at $4.40 per million output tokens. The same setup is said to reach about 3,000 input tokens per second during prefill. Input tokens are cheaper at about $1.40 per million for GLM 5.2, but prefill can be 10 to 30 times faster than decoding, and about 50 times faster in this example. That means output speed alone may understate the value of local hardware for agents that read large inputs.
A personal local AI server is built with four 16GB graphics cards. The hardware is stacked on a kitchen-style shelf instead of a standard server rack, with the main slot split and a riser cable used to connect the cards. The setup runs two llama.cpp instances at the same time and uses a Qwen model in a smaller, lower-memory format. Each instance is set up for about 150,000 tokens of context, with input processing around 1,000 tokens per second and answer generation around 45 to 60 tokens per second. The machine uses an i5 processor and 32GB of DDR4 memory, but the model work mainly runs in graphics card memory. opencode was used to build a backend that manages llama.cpp and counts tokens. A rough cost check suggests about $60 has already been saved. The system is still buggy, and the next step is a router that sends different parallel jobs to different sets of cards and servers.