# \[ Dataset.from\_generator \] Prevent caching during upload

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdataset-from-generator-prevent-caching-during-upload%2F86617%3E%5C
**Category:** 🤗Datasets\
**Created:** [May 14, 2024, 8:26am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdataset-from-generator-prevent-caching-during-upload%2F86617 "2024-05-14T08:26:11Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![jerome-white](https://avatars.discourse-cdn.com/v4/letter/j/a587f6/32.png) [@jerome-white](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjerome-white)%5C
**Post date:** [May 14, 2024, 8:26am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdataset-from-generator-prevent-caching-during-upload%2F86617%2F1 "2024-05-14T08:26:11Z")

</div>

Is it possible to disable caching when using `Dataset.from_generator` ([docs here](https://huggingface.co/docs/datasets/v2.19.0/en/package_reference/main_classes#datasets.Dataset.from_generator))?

I like the method because it doesn’t require me to keep the entire dataset in memory. However, it’s caching so much that I run out of disk before my upload is complete.

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flhoestq)%5C
**Post date:** [May 15, 2024, 12:21pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdataset-from-generator-prevent-caching-during-upload%2F86617%2F2 "2024-05-15T12:21:37Z")

</div>

A `Dataset` always caches the full dataset on disk. If you want to stream the data you should use an `IterableDataset`.

However `IterableDataset` doesn’t implement `push_to_hub()` yet, it would be amazing to add this method for cases like yours

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/de4155eb4aa4108ecb32a1389d7cc37ae69f88b7.png) [@system](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsystem)%5C
**Post date:** [June 20, 2024, 6:48pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdataset-from-generator-prevent-caching-during-upload%2F86617%2F3 "2024-06-20T18:48:27Z")

</div>

Closed on June 20, 2024, 6:48pm UTC

This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.