# How were the GPT2 pretrained tensorflow models created?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-were-the-gpt2-pretrained-tensorflow-models-created%2F357%3E%5C
**Category:** 🤗Transformers\
**Created:** [July 17, 2020, 3:04pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-were-the-gpt2-pretrained-tensorflow-models-created%2F357 "2020-07-17T15:04:22Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![alexsp](https://avatars.discourse-cdn.com/v4/letter/a/46a35a/32.png) [@alexsp](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Falexsp)%5C
**Post date:** [July 17, 2020, 3:04pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-were-the-gpt2-pretrained-tensorflow-models-created%2F357%2F1 "2020-07-17T15:04:22Z")

</div>

Hello there,  
I wonder how the GPT2 pretained models were created. The original models were checkpointed with the tensorflow 1 API and a substantially different computation graph than the reimplemantion in huggingface transformers? I wonder what you did do get there.  
Have you found a way to adapt the originally published weights?  
Have the openai developers shared WebText with you?  
Have you trained the models on similar data?

Thanks for your help

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsgugger)%5C
**Post date:** [July 20, 2020, 1:41pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-were-the-gpt2-pretrained-tensorflow-models-created%2F357%2F2 "2020-07-20T13:41:27Z")

</div>

I wasn’t on the team when this was done, but I guess it was convert to PyTorch using the [conversion from TF scripts](https://huggingface.co/transformers/converting_tensorflow_models.html) and then converted back to TF2 using the functions in the [convert PyTorch to TF2 module](https://github.com/huggingface/transformers/blob/master/src/transformers/convert_pytorch_checkpoint_to_tf2.py).

OpenAI did not share WebText with us, and there was no retraining involved.