# How to prevent too many checkpoints with run\_clm.py?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-prevent-too-many-checkpoints-with-run-clm-py%2F19917%3E%5C
**Category:** Beginners\
**Created:** [July 2, 2022, 1:27am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-prevent-too-many-checkpoints-with-run-clm-py%2F19917 "2022-07-02T01:27:03Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![JacquesThibs](https://avatars.discourse-cdn.com/v4/letter/j/278dde/32.png) [@JacquesThibs](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJacquesThibs)%5C
**Post date:** [July 2, 2022, 1:27am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-prevent-too-many-checkpoints-with-run-clm-py%2F19917%2F1 "2022-07-02T01:27:03Z")

</div>

I’m fine-tuning GPT-2 on a lot of data and ended up using all of my disk space on google drive. I’m not sure which args prevent too many checkpoints. Right now I’m getting a new checkpoint at every 500 example, but I’d like to avoid making so many checkpoints. At the very least only keep the best checkpoint.

Next fine-tuning run I’ll be using the following code (waiting to re-gain access to colab GPU), but I’m not sure if it’ll prevent the additional checkpoints:

```auto
!python gpt-2/run_clm.py \
    --model_name_or_path gpt2 \
    --train_file alignment_texts_87606.csv \
    --do_train \
    --fp16 \
    --overwrite_cache \
    –-overwrite_output_dir \
    -–num_train_epochs 1 \
    --per_device_train_batch_size=2 \
    --output_dir gpt-2/tmp/alignment-texts-clm

```

I don’t think it will. I need something to limit the number of checkpoints. I don’t even know where the number 500 for each checkpoint comes from.

Thanks for the help!

---

<div class="post-metadata">

**Author:** ![BramVanroy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bramvanroy/32/47360_2.png) [@BramVanroy](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FBramVanroy)%5C
**Post date:** [July 3, 2022, 11:09am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-prevent-too-many-checkpoints-with-run-clm-py%2F19917%2F2 "2022-07-03T11:09:19Z")

</div>

You can use `save_total_limit` as an argument. Note that this will just delete older checkpoints, not necessarily the worst ones!

> <https://github.com/huggingface/transformers/blob/49c8c67fb815a277405f84dea4a66353e19fb347/src/transformers/training_args.py#L232-L234>

---

<div class="post-metadata">

**Author:** ![JacquesThibs](https://avatars.discourse-cdn.com/v4/letter/j/278dde/32.png) [@JacquesThibs](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJacquesThibs)%5C
**Post date:** [July 3, 2022, 4:43pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-prevent-too-many-checkpoints-with-run-clm-py%2F19917%2F3 "2022-07-03T16:43:22Z")

</div>

How would I make sure to keep the best model?

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsgugger)%5C
**Post date:** [July 4, 2022, 1:00pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-prevent-too-many-checkpoints-with-run-clm-py%2F19917%2F4 "2022-07-04T13:00:07Z")

</div>

You need to pass along `--load_best_model_at_end` then, so the `Trainer` keeps track of the best model and does not delete it.

---

<div class="post-metadata">

**Author:** ![yubi-sanprit](https://avatars.discourse-cdn.com/v4/letter/y/3ec8ea/32.png) [@yubi-sanprit](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fyubi-sanprit)%5C
**Post date:** [November 7, 2022, 7:59am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-prevent-too-many-checkpoints-with-run-clm-py%2F19917%2F5 "2022-11-07T07:59:37Z")

</div>

@sgugger @BramVanroy @JacquesThibs How can we implement lazy loading of data in RAM while training the model from scratch?

---

<div class="post-metadata">

**Author:** ![yubi-sanprit](https://avatars.discourse-cdn.com/v4/letter/y/3ec8ea/32.png) [@yubi-sanprit](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fyubi-sanprit)%5C
**Post date:** [December 14, 2022, 5:35am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-prevent-too-many-checkpoints-with-run-clm-py%2F19917%2F6 "2022-12-14T05:35:07Z")

</div>

Transformers dataset dict format and its map method to call any function like tokenisation and grouping is designed to run in batches.It will handle any big data with batch run. So, work with any size big data use convert your dataset in Transformers dataset dict format and map method