A search for EMA teacher methods on LoRA adapters
The focus is whether EMA has been used successfully with . The proposed setup would let an EMA adapter act like a teacher and create for the adapter that is still being trained.
A cited on-policy paper uses EMA for the teacher, but it appears to the whole model rather than only . The open question is whether there are empirical results showing this idea works with LoRA or left-to-right models.
Key points
- The item asks for successful uses of EMA with .
- The idea is to use an EMA adapter as a teacher for the adapter being trained.
- The cited work uses EMA, but seems to the full model.
- The main missing piece is empirical evidence for LoRA-based setups.