# Recommended max size of dataset?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Frecommended-max-size-of-dataset%2F144812%3E%5C
**Category:** 🤗Datasets\
**Created:** [March 8, 2025, 9:41pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Frecommended-max-size-of-dataset%2F144812 "2025-03-08T21:41:33Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Aceticia](https://avatars.discourse-cdn.com/v4/letter/a/7c8e57/32.png) [@Aceticia](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FAceticia)%5C
**Post date:** [March 8, 2025, 9:41pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Frecommended-max-size-of-dataset%2F144812%2F1 "2025-03-08T21:41:33Z")

</div>

I’m about to create a large dataset directly, about ~1B samples with each sample being about [16 x 8000] size and some small meta data. Do you foresee any issues during generation, or loading this and using it after it’s finished generating? Any ideas are welcome, thank you.

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [March 9, 2025, 5:01am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Frecommended-max-size-of-dataset%2F144812%2F2 "2025-03-09T05:01:48Z")

</div>

It’s probably going to be over 500TB…

If you’re going to upload more than 300GB of data to Hugging Face in a single repository, it’s better to consult with HF in advance by email. [website@huggingface.co](mailto:website@huggingface.co)

Also, if you’re using a large dataset for training with Hugging Face’s library or torch, it seems that sharding the dataset will make it run more stably. @lhoestq

> [@How to load a large hf dataset efficiently?](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-load-a-large-hf-dataset-efficiently%2F69288)%3A
>
> I am trying to load a dataset [axiong/pmc\_oa · Datasets at Hugging Face](https://huggingface.co/datasets/axiong/pmc_oa) The dataset size is around 22 gb and I have ram ~10 GB, the dataset object is stuck at extracting file point I also tried streaming mode but that’s giving another error. from datasets import load\_dataset dataset = load\_dataset("axiong/pmc\_oa", 'pmc\_oa', split='train', streaming=True) print(next(iter(dataset))) Any suggestion on how to deal with…

> **[Storage limits](https://huggingface.co/docs/hub/storage-limits)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.

---

<div class="post-metadata">

**Author:** ![Aceticia](https://avatars.discourse-cdn.com/v4/letter/a/7c8e57/32.png) [@Aceticia](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FAceticia)%5C
**Post date:** [March 9, 2025, 5:49am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Frecommended-max-size-of-dataset%2F144812%2F3 "2025-03-09T05:49:30Z")

</div>

Hi, thanks for the quick reply! It would be just for training, so upload is not a problem. And I have individual files that I will use `Dataset.from_generator` to create a hf dataset out of, so I think the post you mentioned shouldn’t be a problem either.

I guess I’m more concerned about whether `save_to_disk` would work for something this big, and whether `Dataset.load_from_disk` would be problematic in terms of the number of open files?

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [March 9, 2025, 5:55am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Frecommended-max-size-of-dataset%2F144812%2F4 "2025-03-09T05:55:35Z")

</div>

When it comes to such a huge data set, that’s probably the case…

It’s probably too much for those functions that use the default torch internally, so it might be more stable to use functions related to WebDataset. I think there are other backends or functions that can be used as needed for huge data sets, but I can’t remember…😅

> <https://github.com/huggingface/datasets/issues/5337>
>
> Webdataset is an efficient format for iterable datasets. It would be nice to sup…port it in \`datasets\`, as discussed in https://github.com/rom1504/img2dataset/issues/234.
> 
> In particular it would be awesome to be able to load one using \`load\_dataset\` in streaming mode (either from a local directory, or from a dataset on the Hugging Face Hub). Some datasets on the Hub are already in webdataset format.
> 
> It terms of implementation, we can have something similar to the Parquet loader.
> I also think it's fine to have webdataset as an optional dependency.

> **[WebDataset](https://huggingface.co/docs/hub/datasets-webdataset)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/64813_2.png) [@lhoestq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flhoestq)%5C
**Post date:** [March 11, 2025, 3:22pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Frecommended-max-size-of-dataset%2F144812%2F5 "2025-03-11T15:22:44Z")

</div>

`save_to_disk` / `load_from_disk` can handle big datasets, you can even use multiprocessing with `num_proc=` to accelerate `save_to_disk`

though performance can depend on your environment so I’d still advise you to try on smaller datasets first and see how it scales

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/de4155eb4aa4108ecb32a1389d7cc37ae69f88b7.png) [@system](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsystem)%5C
**Post date:** [March 12, 2025, 5:48pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Frecommended-max-size-of-dataset%2F144812%2F6 "2025-03-12T17:48:57Z")

</div>

Closed on March 12, 2025, 5:48pm UTC

This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.
