TorchJD helps PyTorch train with several goals at once
TorchJD is an PyTorch tool for a model when it has several losses at the same time. Those losses can come from multiple tasks, rules the model must follow, extra helper goals, or regularization. The common approach is to combine all losses into one number, such as an average or a weighted mix, and then train with .
Another approach is to calculate a separate gradient for each loss and combine those into one update that tries to reduce each loss, not just the average. The second approach can help when the goals conflict with each other, but it usually uses more memory. TorchJD now includes many known methods from both approaches, so teams can test them by changing only a few lines of code.
It has also been accepted into the PyTorch .
Key points
- TorchJD helps PyTorch models train with several losses at the same time.
- It supports methods that merge losses into one number and methods that handle each loss separately.
- Separate can help when goals fight each other.
- Those methods can use more memory than simpler loss-merging methods.
- The main value is comparing many methods with small code changes.