3M-param model 'installs' new rules at inference without any training

An independent researcher, working alone and self-funded with a single RTX 3090 GPU, released a preprint proposing a new memory design called a fast-weight memory bank. In this setup, the model writes values into a small bank of vector slots during its own forward pass (a single pass of input through the network to produce output), and later reads those values back not as ordinary data but as if they were model weights. Each slot is expanded by a hypernetwork (a network that generates the weights for another network) into a low-rank MLP layer that gets applied directly to the incoming token stream.

The goal is continual learning during inference with no backward pass at all — no test-time training, no separate optimizer step, no cloning of weights, and no growing context window. The experiment used a tiny 3.08-million-parameter DeepSeek-style transformer fitted with an 8-slot memory bank, tested on a keyed multi-turn rule task: each conversation binds two key tokens to a brand-new rule, shows that rule exactly once as a 13-token sequence, then later asks about symbols the model has never seen paired with that rule. Because each turn is a fully separate forward pass, a rule can only carry over between turns by passing through the memory bank.

Random guessing would succeed only 0.008 of the time, and removing the memory bank entirely (as a control) dropped performance to exactly that chance level everywhere. The main finding is that the approach works and generalizes: a single 13-token presentation of a never-trained rule let the model answer unseen queries correctly 0.79 to 1.00 of the time, replicated across two random seeds. The learned rule also survived even when its storage slot was physically evicted.