# \`train\_test\_split\` with IterableDataset

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrain-test-split-with-iterabledataset%2F29851%3E%5C
**Category:** 🤗Datasets\
**Created:** [January 17, 2023, 3:00am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrain-test-split-with-iterabledataset%2F29851 "2023-01-17T03:00:38Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![zpn](https://avatars.discourse-cdn.com/v4/letter/z/bc8723/32.png) [@zpn](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fzpn)%5C
**Post date:** [January 17, 2023, 3:00am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrain-test-split-with-iterabledataset%2F29851%2F1 "2023-01-17T03:00:38Z")

</div>

Hi all,

Is it possible to use or add a feature to `IterableDatasets` to have a `train_test_split`, similar to the feature [here](https://huggingface.co/docs/datasets/v2.8.0/en/package_reference/main_classes#datasets.Dataset.train_test_split)?

Currently if there’s no train-test-split specified for a dataset (especially a large one), I would have to update the dataset script manually and hack it such that I could define the first X% of the files as train and the rest of split.

I’m guessing that if it’s not currently possible, it might be tricky to implement?

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flhoestq)%5C
**Post date:** [January 23, 2023, 1:48pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrain-test-split-with-iterabledataset%2F29851%2F2 "2023-01-23T13:48:24Z")

</div>

Yes it’s not implemented right now but it should be possible to implement a train\_test\_split over the dataset shards. Contributions are welcome though if you’re interested in helping on this matter 🙂

For now I’d suggest you to define two separate datasets, one with the train data files and one with the test data files

---

<div class="post-metadata">

**Author:** ![zpn](https://avatars.discourse-cdn.com/v4/letter/z/bc8723/32.png) [@zpn](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fzpn)%5C
**Post date:** [January 26, 2023, 4:40am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrain-test-split-with-iterabledataset%2F29851%2F3 "2023-01-26T04:40:49Z")

</div>

Makes sense! Currently, the dataset I’ve been using is only one file but I didn’t want to do 5+ mins of processing ahead of time especially if I had to do this on every machine. I ended up adding a dataset loading script to randomly select columns on the fly (a little hacky but works for now)