(yet) Pytorch Impl of Distributed Shampoo

< 목차 >

• Optimizers for Single Processor
  • Vanilla AdamW
  • Shampoo
  • SOAP
• Distributed Optimizer
• Learning Rate Grafting: Transferability of Optimizer Tuning
• References

Optimizers for Single Processor

Vanilla AdamW

Shampoo

SOAP

Distributed Optimizer

Fig.

Fig.

Fig.

Fig.

Fig.

Fig.

Fig.

Learning Rate Grafting: Transferability of Optimizer Tuning

Fig.

References

• Papers
  • Scalable Second Order Optimization for Deep Learning
  • A Distributed Data-Parallel PyTorch Implementation of the Distributed Shampoo Optimizer for Training Neural Networks At-Scale
  • Learning Rate Grafting: Transferability of Optimizer Tuning
• Distributed Shampoo references
  • facebookresearch/optimizers/distributed_shampoo
  • cloneofsimo/zeroshampoo
  • nikhilvyas/SOAP
• megatron distributed optimizer
  • 1
  • 2