# Pre-trained models that weren't trained on Wikipedia?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpre-trained-models-that-werent-trained-on-wikipedia%2F14530%3E%5C
**Category:** Intermediate\
**Created:** [February 8, 2022, 8:18pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpre-trained-models-that-werent-trained-on-wikipedia%2F14530 "2022-02-08T20:18:34Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![jxm](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/jxm/32/29614_2.png) [@jxm](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjxm)%5C
**Post date:** [February 8, 2022, 8:18pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpre-trained-models-that-werent-trained-on-wikipedia%2F14530%2F1 "2022-02-08T20:18:34Z")

</div>

From the BERT paper:

> For the pre-training corpus we use the BooksCorpus (800M words) (Zhu et al., 2015) and English Wikipedia (2,500M words)

Are there any pre-trained transformers like BERT that weren’t trained on Wikipedia?

---

<div class="post-metadata">

**Author:** ![jxm](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/jxm/32/29614_2.png) [@jxm](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjxm)%5C
**Post date:** [February 8, 2022, 8:20pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpre-trained-models-that-werent-trained-on-wikipedia%2F14530%2F2 "2022-02-08T20:20:08Z")

</div>

from ELECTRA paper:

> For most experiments we pre-train on the same data as BERT, which consists  
> of 3.3 Billion tokens from Wikipedia and BooksCorpus (Zhu et al., 2015). However, for our Large model we pre-trained on the data used for XLNet (Yang et al., 2019), which extends the BERT dataset to 33B tokens …

so that excludes ELECTRA and XLNet too

---

<div class="post-metadata">

**Author:** ![jxm](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/jxm/32/29614_2.png) [@jxm](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjxm)%5C
**Post date:** [February 10, 2022, 2:53am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpre-trained-models-that-werent-trained-on-wikipedia%2F14530%2F3 "2022-02-10T02:53:49Z")

</div>

for posterity: someone recommended “tweet BERT” or something like that, which was only trained on tweets
