# How to save tokenizer after finetunning distilgpt2 model

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-save-tokenizer-after-finetunning-distilgpt2-model%2F15771%3E%5C
**Category:** Beginners\
**Created:** [March 16, 2022, 12:35pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-save-tokenizer-after-finetunning-distilgpt2-model%2F15771 "2022-03-16T12:35:22Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![devpy079](https://avatars.discourse-cdn.com/v4/letter/d/ee7513/32.png) [@devpy079](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdevpy079)%5C
**Post date:** [March 16, 2022, 12:35pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-save-tokenizer-after-finetunning-distilgpt2-model%2F15771%2F1 "2022-03-16T12:35:22Z")

</div>

I am finetuning the distilgpt2 model with my own dataset using clm.

```auto
cmd = '''
python transformers/examples/tensorflow/language-modeling/run_clm.py \
    --model_name_or_path distilgpt2 \
    --train_file {0} \
    --do_train \
    --num_train_epochs 3 \
    --overwrite_output_dir \
    --per_device_train_batch_size 2 \
    --output_dir {1}
'''.format(file_name, weights_dir)

```

But problem is that training is happening successfully but in between this error shows up: Could not locate tokenizer configuration file, will try to use model config file.

The model is saved and config.json is also saved, But when tokenizer is not there, when I tried to load the model it shows tokenizer not found. Can anyone help me to finetunning the distilgpt2 model?

---

<div class="post-metadata">

**Author:** ![merve](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/merve/32/49809_2.png) [@merve](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmerve)%5C
**Post date:** [March 17, 2022, 10:21am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-save-tokenizer-after-finetunning-distilgpt2-model%2F15771%2F2 "2022-03-17T10:21:19Z")

</div>

Hello 👋

Looking at the script you can give

```auto
tokenizer_name: Optional[str] = field(
        default=None, metadata={"help": "Pretrained tokenizer name or path if not the same as model_name"}
    )

```

in the arguments. Otherwise it will use `distilgpt2`'s [own tokenizer config](https://huggingface.co/distilgpt2/blob/main/tokenizer.json). Let me know if this isn’t helpful.

---

<div class="post-metadata">

**Author:** ![devpy079](https://avatars.discourse-cdn.com/v4/letter/d/ee7513/32.png) [@devpy079](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdevpy079)%5C
**Post date:** [March 18, 2022, 11:29am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-save-tokenizer-after-finetunning-distilgpt2-model%2F15771%2F3 "2022-03-18T11:29:49Z")

</div>

Hey @merve thanks for the help! I tried with pytorch based script then it works! will also try your approach with tensorflow script! Also can you help me regarding text generation, when I trained the model, it shows |endoftext| after every result, how to remove it! that is just a marker to the model to stop looking after that!
