A faster simulator helped train a stronger game AI agent

A superhuman AI agent was built for Generals.io, a strategy game with hidden information. The agent reached number one on the public human 1v1 leaderboard and beat the two top-ranked human players in 199 of 269 ladder matches. The first version began as a master’s thesis and used behavior cloning, reinforcement learning fine-tuning, and reward shaping, but it still lost reliably to the best players.

The later version rebuilt the full training pipeline in JAX and replaced a convolutional neural network with a vision transformer. The main idea was to spend effort on scaling the training system instead of adding more hand-made rules and patches. A JAX-based simulator became about 10,000 times faster than the earlier simulator and could run tens of millions of game frames per second on one GPU.

The final agent was trained for four days on four Nvidia H200 GPUs. The code, agent, and fast simulator are open source, so others can reuse the environment for experiments with strategy agents.