# Uploading json, jsonl files as subset on dataset repo
**URL:**
**Author:**  [@sleeping4cat](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsleeping4cat)%5C
**Post date:** [November 12, 2024, 10:57pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fuploading-json-jsonl-files-as-subset-on-dataset-repo%2F117746%2F1 "2024-11-12T22:57:03Z")
I wanted to upload json, jsonl, zip files as subsets but now I know how to upload parquet as subset files. is there any way using API or cli where I could upload any file-type as subset in a dataset repo?
---
**Author:**  [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [November 13, 2024, 2:23pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fuploading-json-jsonl-files-as-subset-on-dataset-repo%2F117746%2F2 "2024-11-13T14:23:44Z")
It seems that the only way to create a subset of a dataset with a normal file is to write a script. However, this forum post is two years old, so the situation may have changed.
> [@How to create subset when pushing to hub](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-create-subset-when-pushing-to-hub%2F19542)%3A
>
> Hey! I have a dataset of image and text, and I am trying to upload it to the hub using the script below. I was wondering how to create a subset, because everything is been put in “gabrielsantosrv–pracegover” subset. Thanks in advance smiley from datasets import Dataset, load\_dataset from PIL import Image import io import json def try\_load\_image(filepath): try: with open(filepath, 'rb') as f: image = Image.open(io.BytesIO(f.read())) if isinstance(image, Image…
---
**Author:**  [@lhoestq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flhoestq)%5C
**Post date:** [November 30, 2024, 4:09pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fuploading-json-jsonl-files-as-subset-on-dataset-repo%2F117746%2F3 "2024-11-30T16:09:12Z")
Hi ! There is no direct API for that yet, but you can set the subset programmatically by editing the Dataset Card data (aka the YAML at the top of the README.md).
See what the YAML should look like if you want multiple configs here: [Manual Configuration](https://huggingface.co/docs/hub/datasets-manual-configuration)
See how to manipulate Dataset Cards here: [Repository Cards](https://huggingface.co/docs/huggingface_hub/v0.26.3/en/package_reference/cards)
For example:
```python
from huggingface_hub import DatasetCard
dataset_card = DatasetCard.load("username/dataset_name")
dataset_card.data["configs"] = [
{"config_name": "subset0", "data_dir": "subset0"},
{"config_name": "subset1", "data_dir": "subset1"},
]
dataset_card.push_to_hub("username/dataset_name")
```
which creates two subsets “subset0” and “subset1” for a dataset structured like this
```auto
my_dataset/
├── README.md
├── subset0/
│ ├── abc.jsonl
│ └── def.jsonl
└── subset1/
└── ghi.jsonl
```
---
**Author:**  [@system](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsystem)%5C
**Post date:** [December 4, 2024, 3:07pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fuploading-json-jsonl-files-as-subset-on-dataset-repo%2F117746%2F4 "2024-12-04T15:07:01Z")
Closed on December 4, 2024, 3:07pm UTC
This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.