# Datasets: Limit the number of rows?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdatasets-limit-the-number-of-rows%2F15747%3E%5C
**Category:** Beginners\
**Created:** [March 16, 2022, 12:33am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdatasets-limit-the-number-of-rows%2F15747 "2022-03-16T00:33:12Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![eddieparker](https://avatars.discourse-cdn.com/v4/letter/e/4bbf92/32.png) [@eddieparker](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Feddieparker)%5C
**Post date:** [March 16, 2022, 12:33am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdatasets-limit-the-number-of-rows%2F15747%2F1 "2022-03-16T00:33:12Z")

</div>

I’m trying to make sure my script I’m hacking works from end-to-end, and waiting for epochs to end in training just takes up a bunch of time. I’ve shortened down the number of epochs and batch size to 1, but I’m guessing the data that I’m using is just too large so it takes a long time to go through batches.

I’m using some code from the GLUE example and it does the following:

```
dataset = datasets.load_dataset("glue", task)

```

I’d like to have it only take a set number of samples so I can iterate quicker.

Some things I’ve tried:

```auto
dataset = datasets.load_dataset(glue, g_task, split=split)
dataset = dataset[:20]|

```

This complains with:

```auto
KeyError: "Invalid key: slice(None, 20, None). Please first select a split. For example: `my_dataset_dictionary['train'][slice(None, 20, None)]`. Available splits: ['test', 'train', 'validation']"

```

Fair, so it’s a dictionary. I then try this:

```auto
	dataset = datasets.load_dataset("glue", g_task, split=split)
	for k, v in dataset.items():
		dataset[k] = v[:20]

```

But then further on things blow up because I’m indexing unexpectedly:

```auto
Traceback (most recent call last):
  File "D:\dev\valve\source2\main\src\devtools\k8s\dota\toxic-chat-ml\test\run_text_classification.py", line 157, in <module>
    train_and_save()
  File "D:\dev\valve\source2\main\src\devtools\k8s\dota\toxic-chat-ml\test\run_text_classification.py", line 55, in train_and_save
    print(f"Sentence: {dataset['train'][0][sentence1_key]}")
KeyError: 0

```

Which makes sense - if I see the two versions of the dict before/after I slice it looks like I’m stomping some metadata with just the output of the training array.

I then see that there’s a “split” command I can issue to load\_dataset that will let me do a slice, but it seems to only work if I request a specific ‘split’ (train/test) and won’t play nice with this dictionary based approach.

Am I missing an alternative here?

Thanks.

-e-

---

<div class="post-metadata">

**Author:** ![eddieparker](https://avatars.discourse-cdn.com/v4/letter/e/4bbf92/32.png) [@eddieparker](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Feddieparker)%5C
**Post date:** [March 16, 2022, 1:07am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdatasets-limit-the-number-of-rows%2F15747%2F2 "2022-03-16T01:07:44Z")

</div>

In case others can use the info, I worked around this by switching to my own dataset and just cutting down the amount of data in it. Crude, but effective.

If there is a way to do this via code however, that would be appreciated. It’d be nice not having to muck with my inputs when I just want to debug something.

---

<div class="post-metadata">

**Author:** ![marshmellow77](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/marshmellow77/32/3279_2.png) [@marshmellow77](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmarshmellow77)%5C
**Post date:** [March 16, 2022, 3:23pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdatasets-limit-the-number-of-rows%2F15747%2F3 "2022-03-16T15:23:28Z")

</div>

Hi Eddie

The datasets documentation shows how to limit the datasets in various ways: [Load](https://huggingface.co/docs/datasets/loading#slice-splits)

Hope that helps 🙂

Cheers  
Heiko

---

<div class="post-metadata">

**Author:** ![eddieparker](https://avatars.discourse-cdn.com/v4/letter/e/4bbf92/32.png) [@eddieparker](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Feddieparker)%5C
**Post date:** [March 16, 2022, 6:07pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdatasets-limit-the-number-of-rows%2F15747%2F4 "2022-03-16T18:07:01Z")

</div>

In case it helps others, this is the solution I went with:

```auto
		split='train'
		if is_dev_run():
			split = 'train[0:100]' # reduce the working size to speed up iteration

		dataset = datasets.load_dataset(input_files_directory, task, split=split)

		# 90% train, 10% test + validation
		train_testvalid = dataset.train_test_split(test_size=0.1)

		# Split the 10% test + valid in half test, half valid
		test_valid = train_testvalid['test'].train_test_split(test_size=0.5)

		# gather everyone if you want to have a single DatasetDict
		dataset = datasets.DatasetDict({'train': train_testvalid['train'], 'test': test_valid['test'], 'validation': test_valid['train']})

```

---

<div class="post-metadata">

**Author:** ![mediocredev](https://avatars.discourse-cdn.com/v4/letter/m/ba8739/32.png) [@mediocredev](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmediocredev)%5C
**Post date:** [December 17, 2023, 8:45pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdatasets-limit-the-number-of-rows%2F15747%2F5 "2023-12-17T20:45:25Z")

</div>

You can use dataset.select(range(10)).

> **[Process](https://huggingface.co/docs/datasets/process)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.