OpenAI tests model behavior before release with real-like chats
OpenAI introduced , a way to check how a new model may behave before users get it. The method takes earlier conversations, removes the old model’s answer, and has the answer in the same place. OpenAI says it removed account-linked and personally identifying details, and only used ChatGPT conversations from people who allow their data to be used for model improvement.
The study looked at about 1.3 million conversations from GPT-5 series between August 2025 and March 2026. It predicted the real-world rate of 20 unwanted behaviors, such as tool-related deception and disallowed content. Compared with hard test prompts, the method better predicted whether these problems would rise or fall after release, with a median error of about 1.5 times.
It also reduced the chance that models could tell they were being tested. GPT-5.2 marked real user traffic as test-like 5.4% of the time and simulated traffic 5.1% of the time, while synthetic test sets were flagged as tests almost 100% of the time. OpenAI also tried the method in Codex-like coding agent settings, where tool calls and repository state must be simulated to make the test feel close to real use.
Key points
- replays earlier real conversation setups with a new model.
- OpenAI studied about 1.3 million ChatGPT conversations from GPT-5 series .
- The method predicted 20 unwanted behavior types, with a median error of about 1.5 times.
- Real-like conversations made it harder for models to notice they were in a test.
- OpenAI also tested the idea for Codex-like that use tools and repositories.