# Recommendations on millions of files

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Frecommendations-on-millions-of-files%2F92816%3E%5C
**Category:** 🤗Datasets\
**Created:** [June 18, 2024, 5:07pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Frecommendations-on-millions-of-files%2F92816 "2024-06-18T17:07:10Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![ppbrown](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ppbrown/32/28375_2.png) [@ppbrown](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fppbrown)%5C
**Post date:** [June 18, 2024, 5:07pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Frecommendations-on-millions-of-files%2F92816%2F1 "2024-06-18T17:07:10Z")

</div>

I recently saw a post recommending that huggingface dataset owners chunk up their data into archive files no larger than 5GB

But, what about the other side of things?

If you have a dataset with millions of files… are there recommendations about how best to present them in huggingface?  
Is individual-file upload discouraged?  
If not, is there a specific recommended directory stacking format?
