5 lessons from running an AI agent used by 45,000 people
A developer's has now been used by 45,000 people, handling 7.5 million messages and 735,000 calls. Running it at that scale produced several lessons.
First, reading actual conversation logs (d before review) is irreplaceable — real users surface countless situations that and QA simply never catch, and reading how the agent handled them is what actually drives decisions about what to fix, add, or leave alone. Second, managing an agent turns out to be a lot like managing a person: giving direction such as reaching back out to people who've gone quiet, handling certain requests differently, or softening the tone, then checking the agent's work in the logs afterward.
Some direction applies universally to every interaction, while other direction is specific to a single customer, and getting both right — continuously, since the work never really finishes — is the core of the job. Third, some users try to push the interaction toward something romantic, and this can only be deterred so much (the source text cuts off here, so the full mitigation approach isn't available).
Key points
- The has served 45,000 people cumulatively, handling 7.5 million messages and 735,000 calls
- Reading d real conversation logs catches issues and QA miss
- Managing an agent resembles managing a person — mixing universal rules with customer-specific direction
- The cycle of give direction → agent acts → check logs repeats continuously and never fully finishes
- Some users try to push interactions toward romantic territory, which can only be partially deterred