(WIP) How Mixtral is trained efficiently?

< 목차 >

• Mixtral Overview
  • Megablocks
• Reference

Mixtral Overview

Fig.

Fig.

Fig.

Megablocks

Fig. Fig.

Fig.

Fig.

Fig.

Fig.

Fig.

Fig.

Reference

• Papers
  • MegaBlocks: Efficient Sparse Training with Mixture-of-Experts
    • stanford-futuredata/megablocks
  • Mixtral of Experts
    • mistralai/mistral-src
    • open-compass/MixtralKit
• Others
  • Mixture of Experts Explained from huggingface blog
  • tmp
  • tmp2