# Stream Audio Dataset that Can't be moved to Hub

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fstream-audio-dataset-that-cant-be-moved-to-hub%2F30941%3E%5C
**Category:** 🤗Datasets\
**Created:** [February 1, 2023, 6:52pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fstream-audio-dataset-that-cant-be-moved-to-hub%2F30941 "2023-02-01T18:52:12Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![picheny](https://avatars.discourse-cdn.com/v4/letter/p/77aa72/32.png) [@picheny](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpicheny)%5C
**Post date:** [February 1, 2023, 6:52pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fstream-audio-dataset-that-cant-be-moved-to-hub%2F30941%2F1 "2023-02-01T18:52:12Z")

</div>

I have a large audio dataset that might be much easier to process if it could be streamed. The problem is that I cannot upload it to the hub because of license restrictions; I can only use it locally on our cluster. The only way possible to stream seems to be to upload it to the hub. Is that correct (in which case, I am out of luck…).

Thanks  
Michael Picheny

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/64813_2.png) [@lhoestq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flhoestq)%5C
**Post date:** [February 2, 2023, 3:34pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fstream-audio-dataset-that-cant-be-moved-to-hub%2F30941%2F2 "2023-02-02T15:34:15Z")

</div>

Hi ! You can definitely stream locally. For example using `load_dataset(..., streaming=True)` on local files

---

<div class="post-metadata">

**Author:** ![picheny](https://avatars.discourse-cdn.com/v4/letter/p/77aa72/32.png) [@picheny](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpicheny)%5C
**Post date:** [February 2, 2023, 5:46pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fstream-audio-dataset-that-cant-be-moved-to-hub%2F30941%2F3 "2023-02-02T17:46:24Z")

</div>

I thought you could only use save\_to\_disk and load\_from\_disk to create local files, and that load\_from\_disk does not support streaming…

THanks  
Michael

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/64813_2.png) [@lhoestq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flhoestq)%5C
**Post date:** [February 7, 2023, 6:37pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fstream-audio-dataset-that-cant-be-moved-to-hub%2F30941%2F4 "2023-02-07T18:37:10Z")

</div>

Yes indeed, save\_to\_disk creates local files and load\_from\_disk doesn’t support streaming from those local files (yet ?).

Only `load_dataset` does right now.

---

<div class="post-metadata">

**Author:** ![picheny](https://avatars.discourse-cdn.com/v4/letter/p/77aa72/32.png) [@picheny](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpicheny)%5C
**Post date:** [February 8, 2023, 4:00am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fstream-audio-dataset-that-cant-be-moved-to-hub%2F30941%2F5 "2023-02-08T04:00:36Z")

</div>

Thanks. So unless the data is uploaded, there are no options for streaming. Is that correct?

Best  
Michael

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/64813_2.png) [@lhoestq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flhoestq)%5C
**Post date:** [February 16, 2023, 12:04pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fstream-audio-dataset-that-cant-be-moved-to-hub%2F30941%2F6 "2023-02-16T12:04:27Z")

</div>

You can save the dataset to parquet locally using `to_parquet`, and then reload the parquet data in streaming mode using `load_dataset` 🙂

---

<div class="post-metadata">

**Author:** ![vimey](https://avatars.discourse-cdn.com/v4/letter/v/e9c0ed/32.png) [@vimey](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvimey)%5C
**Post date:** [March 17, 2023, 10:00am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fstream-audio-dataset-that-cant-be-moved-to-hub%2F30941%2F7 "2023-03-17T10:00:28Z")

</div>

I have a similiar problem… @lhoestq Could you give us a bigger code example of the idea you suggested?

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/64813_2.png) [@lhoestq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flhoestq)%5C
**Post date:** [March 17, 2023, 11:17am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fstream-audio-dataset-that-cant-be-moved-to-hub%2F30941%2F8 "2023-03-17T11:17:56Z")

</div>

Sure, here you go 🙂

without writing new files:

```python
ids = ds.to_iterable() # optional pass num_shards= if you want to shuffle later or for parallel loading with a dataloader

```

by writing dataset to parquet and stream it later:

```python
ds.to_parquet("parquet_dir/train.parquet")
# later
ids = load_dataset("parquet_dir", streaming=True, split="train")

```

and if your dataset is big you can even save it in shards:

```python
num_shards = 16
for index in range(num_shards):
    shard = ds.shard(num_shards=num_shards, index=index, contiguous=True)
    shard.to_parquet(f"parquet_dir/train-{index:05d}-of-{num_shards:05d}.parquet")
# later
ids = load_dataset("parquet_dir", streaming=True, split="train")

```
