Question about a LoRA paper figure on subspace similarity
A figure in the LoRA paper raises confusion about what its axes mean. The is understood as measuring how much the made by the top i vectors is contained inside the made by the top j vectors from a higher-rank matrix. Under that reading, j cannot be smaller than i.
The paper says the third and fourth figures zoom in on the lower-left triangle of the two left figures. That seems to create values where j is 1 while i ranges from 2 to 8, so the exact meaning of the y-axis in the two right figures is unclear.
Key points
- The item focuses on a figure in the LoRA paper.
- The is interpreted as comparing top i vectors with top j vectors from a higher-rank matrix.
- That interpretation suggests j should not be smaller than i.
- The confusion comes from the right-side figures, where the y-axis seems hard to reconcile with the stated zoomed area.
- This is more about LoRA theory than a direct cost-saving tactic.