# How to split a dataset into train, test, and validation?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-split-a-dataset-into-train-test-and-validation%2F1238%3E%5C
**Category:** Beginners\
**Created:** [September 23, 2020, 9:23pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-split-a-dataset-into-train-test-and-validation%2F1238 "2020-09-23T21:23:10Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Raffal](https://avatars.discourse-cdn.com/v4/letter/r/5daacb/32.png) [@Raffal](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FRaffal)%5C
**Post date:** [September 23, 2020, 9:23pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-split-a-dataset-into-train-test-and-validation%2F1238%2F1 "2020-09-23T21:23:11Z")

</div>

I am having difficulties trying to figure out how I can split my dataset into train, test, and validation. I’ve been going through the documentation here:

> <https://github.com/huggingface/datasets/blob/37d4840a39eeff5d472beb890c8f850dc7723bb8/datasets/wikihow/wikihow.py>

  
and the template here:  

> <https://github.com/huggingface/datasets/blob/master/templates/new_dataset_script.py#L63>

  
but it hasn’t become any clearer.

this is the error I keep getting:  
TypeError: ‘NoneType’ object is not callable

and this is the code I’m using:

```
def _split_generators(self, dl_manager):
    """Returns SplitGenerators."""
    dl_path = dl_manager.download_and_extract(_URLS)
    titles = {k: set() for k in dl_path}
    for k, path in dl_path.items():
        with open(path, encoding="utf-8") as f:
            for line in f:
                titles[k].add(line.strip())

    path_to_manual_file = os.path.join(
        os.path.abspath(os.path.expanduser(dl_manager.manual_dir)), self.config.filename
    )

    if not os.path.exists(path_to_manual_file):
        raise FileNotFoundError(
            "{} does not exist. Make sure you insert a manual dir via `datasets.load_dataset('wikihow', data_dir=...)` that includes a file name {}. Manual download instructions: {})".format(
                path_to_manual_file, self.config.filename, self.manual_download_instructions
            )
        )
    return [
        datasets.SplitGenerator(
            name=datasets.Split.TRAIN,
            gen_kwargs={
                "path": path_to_manual_file,
                "title_set": titles["train"],
            },
        ),
        datasets.SplitGenerator(
            name=datasets.Split.VALIDATION,
            gen_kwargs={
                "path": path_to_manual_file,
                "title_set": titles["validation"],
            },
        ),
        datasets.SplitGenerator(
            name=datasets.Split.TEST,
            gen_kwargs={
                "path": path_to_manual_file,
                "title_set": titles["test"],
            },
        ),
```

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvalhalla)%5C
**Post date:** [September 24, 2020, 4:22pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-split-a-dataset-into-train-test-and-validation%2F1238%2F2 "2020-09-24T16:22:07Z")

</div>

I think it’s answered here  
[How to split main dataset into train, dev, test as DatasetDict](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-split-main-dataset-into-train-dev-test-as-datasetdict%2F1090)

---

<div class="post-metadata">

**Author:** ![brando](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/brando/32/30114_2.png) [@brando](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fbrando)%5C
**Post date:** [May 17, 2024, 5:17pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-split-a-dataset-into-train-test-and-validation%2F1238%2F3 "2024-05-17T17:17:07Z")

</div>

```auto
from datasets import load_dataset, DatasetDict

# Load a dataset from Hugging Face
dataset = load_dataset('squad', split='train')

# Split the dataset into training and validation sets
# Specify the fraction for the test set (validation set)
train_val_split = dataset.train_test_split(test_size=0.1)

# Extract the training and validation datasets
train_dataset = train_val_split['train']
val_dataset = train_val_split['test']

# Print the size of the datasets
print(f"Training set size: {len(train_dataset)}")
print(f"Validation set size: {len(val_dataset)}")

# Save the datasets if needed
# train_dataset.save_to_disk('path/to/train_dataset')
# val_dataset.save_to_disk('path/to/val_dataset')

```