# Limitations of iterable datasets

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Flimitations-of-iterable-datasets%2F16794%3E%5C
**Category:** 🤗Datasets\
**Created:** [April 13, 2022, 1:54pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Flimitations-of-iterable-datasets%2F16794 "2022-04-13T13:54:45Z")\
**Posts on this page:** 1\
**Showing post:** 2

<div class="post-metadata">

**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmariosasko)%5C
**Post date:** [April 21, 2022, 11:58am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Flimitations-of-iterable-datasets%2F16794%2F2 "2022-04-21T11:58:40Z")

</div>

Hi!

> \_ the map function of iterable datasets doesnt seem to accept the num\_proc argument, I wonder whether this will create a bottleneck in my codes or if dataloader\_num\_workers will allow the iterable dataset to operate in fast multi-processing ?

Adding support for multiple workers (`num_workers > 1`) to `IterableDataset` is a work in progress and will be available (most likely) in the next release of `datasets`. But in your case, for maximum performance, it’s better to use the standard arrow-backed `Dataset`. Thanks to memory mapping, this version also doesn’t bring everything in memory (only the requested rows/columns).

You can create a dataset from parquet files (the arrow backed version) as follows:

```python
from datasets import load_dataset
dataset = load_dataset("parquet", data_files=[<list of paths to parquet files>])

```

> \_ when working in run\_mlm.py with the trainer and an iterable dataset, what are the changes to make for parallel-processing please ?  
> I read this [Process](https://huggingface.co/docs/datasets/master/process#distributed-usage) but I am not sure if this applies

You can use the `training_args.main_process_first` context manager for that (for the arrow backed dataset). You can find an example [here](https://github.com/huggingface/transformers/blob/705d65368fb28246534ef636fe62c008f4fb2682/examples/pytorch/language-modeling/run_mlm.py#L419).

> \_ my datasets are stored as .parquet containing input sequences as well as labels/meta-data, one column I would like to implement is a sampling probability in order to over-sample certain training examples  
> Is there any way to allow this inside an iterable dataset or should I consider duplicating training examples as a pre-processing ?

I’m not sure I understand this question. Could you clarify it a bit more?

---

_[View the full topic](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Flimitations-of-iterable-datasets%2F16794)._