Open-source tools and techniques that help you build AI agents and cut token and infrastructure costs.
A recorded call needs more than speech recognition if the result must show what was said, who said it, and when. Open speech recognition tools such as Whisper, Parakeet, and Voxtral usually produce words, but they do not reliably add speaker labels on their own. WhisperX is a common open-source pipeline for doing the full job in one place. It uses faster-whisper to transcribe speech with rough timing, wav2vec2 forced alignment to tighten timing down to individual words, pyannote to split the audio by speaker, and then matches the word timeline to the speaker timeline. Real meeting audio is much harder than clean benchmark audio. Whisper large-v3 has about a 2% word error rate on clean LibriSpeech audio, but about 16% on AMI meeting audio, which is roughly eight times worse. Whisper can also invent words during quiet parts, so VAD and condition_on_prev_text=False help reduce that problem. Overlapping speakers, crosstalk, and far-away microphones are major reasons speaker labeling fails.
A RAG system can expose private data during evaluation, not only during setup. A simple test is to disconnect the network and run the evaluation suite. Any part that fails with a connection error was likely calling an outside model to grade the result. In many RAG evaluations, that grader sees the user question, the retrieved document chunks, and the generated answer together. Remote embedding APIs can send documents outside the network when the index is built, but hosted judge models can also send sensitive data outside every time evaluation runs. This may be acceptable for a chatbot built on public documents. It is much riskier for contracts, patient notes, internal source code, or customer support tickets. The suggested approach is to use evaluation code that can be read, forked, and run inside the network under the Apache-2.0 open-source license.
Large organizations have contracts, rules, technical documents, and manuals with sections, definitions, and cross-references, but those documents are not easy for software to use reliably. An LLM can pull out entities, relationships, and rules from this kind of text, but that is only the first step. The same document can produce different results if the model, prompt, or surrounding text changes. The ontology also changes over time when new document types appear, experts correct the design, or other teams need different data shapes. Extracted data should be checked before storage with schema validation, duplicate checks, conflict checks, and basic sanity checks on values. Every item in the graph should keep provenance, including the source document, model version, extraction time, validation result, and human review status. High-confidence results can move forward automatically, while unclear or conflicting results should go to a human review queue. A verified evaluation dataset is needed so teams can test quality whenever prompts, models, or schemas change.
AI agent memory often focuses on how well the system can find stored information. A more basic problem comes first: knowing when different records belong to the same person. One person may appear as an anonymous browser ID, a logged-in user ID, and a server-side ID. If the memory system treats those as separate people, context saved in one session will not carry over to the next session or another device. Builders need to decide whether memory should be keyed to the logged-in user ID, a hash, or handled by the main app before retrieval matters.
An offline, on-premise n8n workflow is being used to answer questions about a company’s internal documents. The documents are stored in Qdrant as embeddings, and the qwen3 embeddings model creates those embeddings. Ollama serves the models locally. An AI agent node answers user questions with Qwen3-Coder 30B as the chat model. The Qdrant retriever returns relevant information, but the agent does not turn that information into accurate answers. Some answers also include hallucinations. A larger chat model is not practical because the available hardware is limited, so the workflow needs accuracy improvements without moving to a heavier model.
A months-long set of experiments used single-file three.js games that could be built from one prompt. Simple game tasks, such as Minecraft-style clones, were chosen because they are common in training data and easy to check by sight and logs. The goal was not to compare model quality, but to tune the harness, system prompt, and tool calling with tasks that were cheap to repeat. The harder test asked Claude Code on Opus 4.8 and a local Qwen3.6 27B agent to build a small raytraced first-person shooter demo in C, using only the standard library. Both agents struggled to finish it in one shot at first. The result changed after one added requirement: the compiled program needed a headless mode where the agent could send keyboard and mouse input and take a screenshot at a chosen frame.
An LLM judge can approve a response while a human reviewer sees the same response as unsafe. The issue is not always that the LLM judge ignores the written rubric. The deeper problem is that the rubric may miss subtle human judgment. This can include a small policy violation, wording that technically follows the rules but would upset a real customer, or an answer that is correct but wrong for the situation. Putting a human on every evaluation would be too costly and slow, which weakens the reason to use an LLM judge in the first place. The practical question is how to reduce the gap when the LLM judge and human reviewers disagree on safety.
LatentGate is a Python tool that shrinks images, long text, chat history, and documents on a local computer before sending a smaller package to an outside AI model. The local step runs through Ollama, and the smaller result can go to services such as OpenAI, Anthropic, Google, Groq, or an OpenAI-compatible endpoint. The goal is to reduce the number of tokens sent to paid AI services. Its example says a direct image request can use about 1,200 tokens, while the LatentGate flow sends about 200 tokens instead. Its benchmarks also show long text going from about 800 tokens to 120, a 10-turn conversation from about 2,500 to 350, and three documents for a question from about 3,000 to 450. For 10,000 image questions with gpt-4o-mini, it estimates input tokens falling from 12 million to 2 million and cost falling from $1.80 to $0.30. It can run as an MCP server for tools such as Claude Desktop, Cursor, Cline, Continue, and Zed, so those tools can compress images, long prompts, large chats, and document questions automatically. It also includes caching, model preloading, parallel processing, and a video mode that calls the outside AI only when a scene changes.
GLM-5.2’s published benchmark numbers appear close to GPT-5.5 across several tests and not far behind Opus 4.8 in many areas. That would be a notable jump for a Chinese model compared with roughly six months ago. The weaker point is token cost in agentic coding tasks. GLM-5.2 Max is shown using almost twice as many tokens as Opus 4.7 Max for a similar score. Opus 4.8 High still looks much better when score is compared with token use. In practical use, cheaper Chinese models can take over some work to reduce API spending, but Claude is still preferred for harder reasoning and prompts with many conditions. GLM-5.2 may also be slower on larger jobs and use more tokens, so it is better seen as a way to split work across models, not as a full replacement.
The goal is to pull structured data from PPTX and PDF project packs, including activities, milestones, risks, issues, owners, statuses, and dates. The hardest part is the Gantt chart. These charts can look very different from one document to another, with months, quarters, years, week-start labels, fiscal periods, mixed time scales, different bar styles, milestone icons, legends, and layouts. Some files contain real PPTX shapes, while others contain screenshots or flattened PDFs that are harder to inspect directly. A reliable system would likely need a mix of LLM and VLM reasoning, rule-based extraction, OCR, file parsing, and coordinate-based date mapping. The key challenge is getting start dates, end dates, and milestone dates from many chart formats without hardcoding each document style.
A graph-backed retrieval pipeline can fail even when the retriever, the language model, and the query syntax all look correct. The real problem can be semantic: the system follows valid relationships but combines them into a false conclusion. For example, “Person manages Team” and “Team uses Tool” can both be true, but that does not mean the person personally uses the tool. A query engine usually checks whether a query is valid, not whether the combined meaning is valid. Schema validation caught type mismatches, but it did not check whether a person should inherit a team’s tool use through a management relationship. Query logging showed which retrieval steps ran, but not why the final answer was wrong. Asking the language model to check itself did not help, because the retrieved context supported the wrong conclusion and made the model confirm the mistake.
Consumer AI needs to feel less like a one-time tool if people are going to keep coming back to it. Many AI products today work as short transactions: open the app, ask for help, get an answer, finish the task, and leave. That is useful for work, coding, writing, research, planning, and productivity, but it does not create much personal attachment. A stronger product would remember the person, understand how they think, help them think, and become more useful with repeated use. That makes it closer to someone people want to talk with, not just an assistant they use for a task. For frequent and emotionally meaningful use, the AI likely needs to live on the phone, because that is where people already spend much of their time. A browser tab or desktop tool may feel too distant for this kind of personal product.
Anthropic and similar AI companies are criticized for using “AI safety” to justify limits that may also protect money and control. The argument is that major AI products were built on open-source tech, public research, and knowledge produced by many people online. After gaining power, these companies decide who gets serious access, who can build, who can research, who stays in basic consumer mode, and who is pushed out by price. Safety is still treated as important, but the concern is that the word can hide business control when used too broadly. The result, in this view, is a tool built from public knowledge being sold back as an expensive closed product with guardrails.
A RAG demo worked well at first, but answer quality dropped when real users started using it. Plain vector search was not enough because text chunks that looked similar in meaning were often wrong on the facts. Hybrid search, combining BM25 with dense search, plus a reranking step improved results more than switching models. Chunking had a bigger impact than model choice. Fixed-size chunks damaged tables and code, while splitting documents by their structure fixed many of those problems. Without evaluation, there was no reliable way to know whether changes helped. A golden dataset made it possible to measure retrieval precision after every change. About half of the changes that seemed better were actually regressions, and most gains came from retrieval engineering rather than prompt changes.
Astraea.cpp and Astraea for Python are being built as a tenant-facing question-and-answer tool for New Zealand tenancy law. The system uses a legislation-first RAG design. It indexes the Residential Tenancies Act, Healthy Homes Standards, Tenancy Tribunal decisions, and official Tenancy Services guidance. It retrieves legislation, official guidance, and cases as separate source types instead of mixing them all together. Important law sections are routed by fixed rules, and vector search is used as a backup signal when no rule matches but legislation retrieval looks confident. A local large language model writes answers with citations. The system also shows the context and debug output so the builder can see what the model actually used. There is also a dataset of 300 verified real-world tenancy question-and-answer pairs, but those answers are being treated as a practical advice floor rather than the final legal gold standard because they do not always include law sections or Tribunal citations.
Local models were often treated like toys about a year ago, mainly useful for private chat, simple conversations, or small document search tasks. They are now being used for coding, private documents, local workflows, and replacing some paid API calls. Models such as Gemma, Qwen, GLM, and Kimi are part of this shift. They still do not fully replace the best closed models for long work across large code repositories. The gap is still clear when a task needs planning, broad context, and the ability to find and fix its own mistakes. The improvement likely comes from better base models, better quantization, stronger tools such as llama.cpp and Ollama, and more available VRAM.
Using Cursor and Claude for almost all of a basic side project can feel very fast at first. The first setup work can be finished in about an hour. As the project grows, more time can shift into reading long context, rewriting prompts, and checking why the AI keeps making the same mistake. The work changes from fixing syntax errors to supervising an AI that apologizes but still repeats bad output. A more productive setup may be to write core logic by hand and use AI for repeated setup code or routine pieces.
AI agents still face several blockers before wider use: reliability, cost, trust, user experience, and a lack of clearly strong use cases. The strongest concern is reliability. Many company tasks being handed to AI agents are closer to basic automation, where a predictable script may be cheaper and safer. AI agents can become risky when they lack needed information and guess the missing parts, because that can lead to hallucination and failed compliance tests. A more practical path is to give clear instructions, add context from past requests and successful outcomes, and use parameter-driven restrictions to narrow what the agent can do. Another barrier is a knowledge gap: many people do not yet know how to build good agents, and many teams do not yet know which jobs actually need them.
Building AI agents with Claude can make short-term memory feel strong, because a lot of information can fit into one interaction. But token limits and memory are not the same issue. A larger context window lets more information go into a single prompt, but it does not decide what should be kept for later, what should be shortened, what should be forgotten, or which user preferences and decisions should carry across sessions. In practice, many memory problems are really about retrieval and structure, not just input size. Very long prompts can hide weak memory design and make a model seem more capable without making it truly persistent. The hard part is deciding what information deserves to live outside the prompt.
Voice agents work by turning speech into text, sending that text to an AI model, and turning the answer back into spoken audio. Building them is hard because several parts must work together, including speech recognition, a language model, speech synthesis, voice activity detection, turn detection, and phone connections. The learning path starts with the basic speech-to-AI-to-speech flow and the delay that builders need to manage. It then moves to making a simple first project with LiveKit Agents or Pipecat, both open-source options. Later steps show how to swap each part so learners understand what each layer does. The path also covers connecting a real phone number through SIP and preparing a system for production with testing, safety work, and rules such as the FCC and EU AI Act. It includes more than 190 resources, marks commercial sources, sorts materials by level, and ends with a 5-week plan.
AI agents that handle commercial tasks may need more than a language model, tool connections, memory, and workflow systems. They may need a separate discovery API that lets the agent check available offers, supported merchants, regions, categories, business terms, required disclosures, click tracking, conversion tracking, and billing rules. Without this layer, developers may have to connect each merchant by hand, rely on broad affiliate links, or build a separate tracking setup for each workflow. That approach does not scale well as more merchants and use cases are added. Product recommendations could become a normal part of agent commerce, especially for agents that suggest software, work tools, local services, financial products, travel, education, or online shopping options.
Thousands of PDFs need to be turned into a relational table with product numbers and prices. The files do not follow one format: some have clean tables, while others place prices inside ordinary paragraphs. Extra details such as delivery charges and other pricing conditions also affect the final price. Those conditions do not apply to every product, so attaching them to all products would create wrong results. The main options are Amazon Bedrock Data Automation, which fits an AWS-heavy setup, or sending the PDFs to a large language model and asking for structured output with a confidence score. The managed service may be easier to operate, but cost becomes a concern at this document volume. The practical question is which approach works reliably in production, how accurate it is, and how its cost can be controlled.
Sales through AI agents can be more complicated than a simple recommendation and purchase. A person may ask several questions before buying, and the agent may compare several products before suggesting one. The suggestion may happen after a tool call, and the person may click later, switch devices, or buy in a later session. This makes attribution hard because it is unclear who actually influenced the sale. The system also has to know which offer was recommended, whether the recommendation was paid or natural, whether the user was told clearly, which merchant should pay, and which agent or publisher should be rewarded. Fraud and low-quality traffic also need to be handled. Without reliable tracking, merchants may not trust agent-driven sales. The main claim is that agent commerce needs new infrastructure instead of simply reusing standard affiliate links.
AI agents can work once and then behave differently the next time. The difference may be a different tool call, different arguments, an unexpected branch, a loop, or a state problem. A trace may exist, but the hard part is still finding the exact point where the behavior changed. A proposed debugging tool compares a replay with a reference run and highlights the first place where the two runs drift apart. Current ways to handle this include LangSmith, Langfuse, evals, custom logs, and manual trace comparison.
Search ads are built around a simple pattern: a person types a query, a results page appears, ads compete for visible spots, clicks are measured, and later sales or sign-ups are linked back to the ad. AI agents can break that pattern. A person’s intent may appear in a chat, a search box, a task request, a tool call, or a recommendation flow. A business result may not come from a clear ad slot. It may come from a recommendation, a comparison, a tool result, or a discount hidden inside a larger task. That means the supporting system has to change too. Ranking alone is not enough; disclosure, quote metadata, tracking, settlement, and merchant reports may all matter. AI-based business discovery could become a new mix of search intent, assistant recommendations, affiliate marketing, and merchant reporting.
A real Q3 strategy brief was tested with Minimax M3 using 14 sources. The input included PDFs, earnings call transcripts, and two analyst notes, totaling about 340,000 tokens. The goal was to create one strategy document while keeping a clear map between each claim and its source. Source tracking stayed useful across the long input. The model could separate whether a claim came from a Gartner note or a competitor earnings call without extra prompting. The weak point appeared after roughly 200,000 tokens. Below that level, caveats and limits stayed visible; above it, the model tended to smooth over conflicts instead of marking them clearly. Minimax M3’s stated BrowseComp 83.5 result and ICLR replication example may show other strengths, but they do not directly prove reliability on long business synthesis work.
ROS-1 Lite is an early public tool that uses GPT to evaluate the structure of AI answers. Its main idea is that many answers are not just right or wrong. Some answers have enough support to reach a conclusion, some should stay open because key information is missing, and some should keep several possible explanations alive. ROS-1 Lite sorts answers into states such as RESOLVED, OPEN, SUPERPOSED, and COLLAPSED. It also checks whether an answer has too little support, adds more structure than the evidence can carry, or uses impressive new terms that do not help the judgment. It does not promise to find the truth, and it does not replace expert review. Its purpose is to show what supports the conclusion, what is still missing, and whether an answer closed the question too early.
patchright-cli is a small open-source tool that lets AI agents open and control real Google Chrome from the command line. Common Playwright or CDP browser setups can be detected and blocked by some websites. This tool uses Patchright underneath to reduce that problem. The basic flow is simple, with commands such as `open`, `snapshot`, `click`, and `close`. playwright-cli and patchright-cli are patched to work as interchangeable tools, so an agent does not need to decide which one to use. Because it runs a visible, real Chrome browser, it can reach many sites that block generic browser automation tools, though it is not perfect. It also includes simple hints that tell agents to close heavy tabs, because agents may otherwise leave many tabs open and waste system resources.
Claude Desktop’s Code mode is useful for code work because it shows code clearly and supports MCP-based browser actions and screenshot checks. The practical goal is to keep that interface while sending model requests to a local API instead of relying only on a hosted model. The key question is where Claude Desktop can be configured to point at local language model APIs. Setup help for Ollama, llama.cpp, or vLLM is not needed, so those local model servers are assumed to be ready already.
H64LM is a 249-million-parameter mixture-of-experts model built directly in PyTorch. Its goal is to help study how modern large language models work, not to provide a strong ready-to-use model. The core parts were written manually instead of using a high-level training framework, including attention, expert routing, normalization, and the training loop. The design includes grouped query attention, a sparse mixture-of-experts setup with 8 experts and top-2 routing, three helper losses for routing, SwiGLU, RoPE, RMSNorm, and sliding-window attention. The training setup includes mixed-precision training, gradient accumulation, a custom training loop, checkpoint saving, and resume support. The released checkpoint was trained on part of WikiText-103 only to prove the pipeline works end to end; it is visibly overfit after epoch 10, with a best validation perplexity of about 40.5. Current limits include generation with batch size 1 only and no true DDP, with DataParallel used instead.