QLoRA’s 2e-4 default may be too high for datasets under 10k
QLoRA guides from Unsloth and , along with the original paper, commonly use 2e-4, or 0.0002, as a starting . That value came from Alpaca training with about 52,000 examples, while many small custom datasets contain only 5,000 to 10,000. With less data, can begin during the first epoch: training loss keeps falling, but loss stays flat or rises.
In a firsthand experiment, about 8,000 rows were cleaned down to roughly 7,200; two rounds of data cleaning, two prompt-template rewrites, and manual relabeling still produced seven similarly poor . Lowering the from 2e-4 to 1e-4 and increasing training from three epochs to five improved more than all the earlier changes combined. Three additional runs showed the same pattern.
For datasets below 10,000 examples, 2e-4 should therefore be tested as a starting guess rather than treated as a fixed answer.
Key points
- The 2e-4 setting originated from Alpaca training with about 52,000 examples.
- Datasets with only 5,000 to 10,000 examples may begin in the first epoch.
- With roughly 7,200 rows, the combination of a 1e-4 and five epochs beat the earlier 2e-4 and three-epoch setup.
- Watch loss rather than assuming that falling training loss means the model is improving.
- Test and epoch count separately to learn which change actually helps.