# Training speed vs Megatron

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftraining-speed-vs-megatron%2F63989%3E%5C
**Category:** 🤗Transformers\
**Created:** [November 29, 2023, 11:17am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftraining-speed-vs-megatron%2F63989 "2023-11-29T11:17:50Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![xianf](https://avatars.discourse-cdn.com/v4/letter/x/a3d4f5/32.png) [@xianf](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fxianf)%5C
**Post date:** [November 29, 2023, 11:17am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftraining-speed-vs-megatron%2F63989%2F1 "2023-11-29T11:17:50Z")

</div>

I try to pretrain a language model so I made some speed tests about transformers and Megatron. I found if I pre-tokenize the data and use flash-attention, the training speed is on par with Megatron in one node with 8 GPU.  
Is this correct? As I know, transformers is slower than Megatron, especially for pretrain from scratch.