# Efficiently slicing dataset

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fefficiently-slicing-dataset%2F28067%3E%5C
**Category:** 🤗Datasets\
**Created:** [December 18, 2022, 3:00pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fefficiently-slicing-dataset%2F28067 "2022-12-18T15:00:39Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![JustSaX](https://avatars.discourse-cdn.com/v4/letter/j/3be4f8/32.png) [@JustSaX](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJustSaX)%5C
**Post date:** [December 18, 2022, 3:00pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fefficiently-slicing-dataset%2F28067%2F1 "2022-12-18T15:00:40Z")

</div>

Hi,

What is the most efficient method of slicing a dataset? I have a dataset that only contains the input\_ids and attention mask for evaluating my model, both as list (not Tensor). I need to split the dataset into chunks as my GPU Memory is not enough to fit everything in one go.

I tried `ds.select(range(0,20))` and I tried `ds[0:20]`. Both operations take 2 seconds and even If I increase the size from 20 to 50 elements it takes 5 seconds. So basically the time increases linearly with the number of elements.

Is there a way to slice a dataset that is more time efficient?

Thanks!

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/64813_2.png) [@lhoestq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flhoestq)%5C
**Post date:** [December 18, 2022, 11:14pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fefficiently-slicing-dataset%2F28067%2F2 "2022-12-18T23:14:25Z")

</div>

You can use `ds.select()`. Make sure to use a recent version of `datasets`, it has been optimized significantly and is almost instantaneous even for big datasets

---

<div class="post-metadata">

**Author:** ![JustSaX](https://avatars.discourse-cdn.com/v4/letter/j/3be4f8/32.png) [@JustSaX](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJustSaX)%5C
**Post date:** [December 22, 2022, 3:54pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fefficiently-slicing-dataset%2F28067%2F3 "2022-12-22T15:54:20Z")

</div>

Thanks for the help!

The problem was somewhere else in the end. The print statements in my code find out which expression takes the longest slowed down my code substantially 🤦‍♂️.

I learned my lesson 😄