Flash Attention 1 & 2

Distributed training Tag

2025

01-20

Flash Attention 1 & 2

01-18

Megatron-Turing NLG 530B

01-18

Other papers on DeepSpeed-MoE

01-15

DeepSpeed-MoE

01-13

ZeRO-Infinity

01-13

ZeRO-Offload

01-01

ZeRO (DeepSpeed from Microsoft)

2024

12-26

Megatron-LM (3)

12-26

Megatron-LM (2)

12-26

Megatron-LM (1)