(WIP) Rethinking Conventional Wisdom from the LLM perspective

< 목차 >

• tmp
• (23th Sep 2024) Rethinking Conventional Wisdom in Machine Learning: From Generalization to Scaling
  • Weight Decay (and muP)
• References

tmp

(23th Sep 2024) Rethinking Conventional Wisdom in Machine Learning: From Generalization to Scaling

Fig.

Weight Decay (and muP)

Fig.

Fig.

References

• Papers
  • A Theory on Adam Instability in Large-Scale Machine Learning
  • Why Do We Need Weight Decay in Modern Deep Learning?
    • openreview
  • Rethinking Conventional Wisdom in Machine Learning: From Generalization to Scaling
    • tweet
  • L2 regularization versus batch and weight normalization
• tmp
  • Open Pretrained Transformers - Susan Zhang, Stanford MLSys #77
  • tweet from Victor Sanh
  • Knowledge sharing note - What did we learn in the past 3 months?

  • Motsepe-Jr/Challenges-Faced-Pretraining-LLMs

  • tweet thread bout weight decay
  • Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networks
    • tweet
• tmp 2
  • Weight decay exclusions from minGPT issues
  • Why not perform weight decay on layernorm/embedding?
  • Exploring Weight Decay in Layer Normalization: Challenges and a Reparameterization Solution from Ohad Rubin
  • When should you not use the bias in a layer?
  • L2 Regularization versus Batch and Weight Normalization
  • L2 Regularization and Batch Norm