(yet) Param and Activation Norm Growth and Loss Spike

< 목차 >

• tmp
• References

tmp

References

• Papers
  • Spike No More: Stabilizing the Pre-training of Large Language Models
  • ReZero is All You Need: Fast Convergence at Large Depth
• Blogs
  • Investigating Pretraining Dynamics and Stability with OLMo Checkpoints
  • Residual stream norms grow exponentially over the forward pass
  • Re-Examining LayerNorm
  • Basic Facts about Language Model Internals
  • Exploring the Residual Stream of Transformers for Mechanistic Interpretability — Explained
  • A Mathematical Framework for Transformer Circuits