# Datasets caching from\_pandas()

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdatasets-caching-from-pandas%2F170648%3E%5C
**Category:** 🤗Datasets\
**Created:** [November 19, 2025, 3:04pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdatasets-caching-from-pandas%2F170648 "2025-11-19T15:04:22Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![jwidmer](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/jwidmer/32/52233_2.png) [@jwidmer](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjwidmer)%5C
**Post date:** [November 19, 2025, 3:04pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdatasets-caching-from-pandas%2F170648%2F1 "2025-11-19T15:04:22Z")

</div>

Hello everyone  
I’m handling a quite big dataset locally - trying to save it `to_parquet()` after some processing.  
Next to some string/int columns, there is one containing images.

I decided to follow those steps:

1. Reading the data from a ZIP or folder in a `pandas.DataFrame`. The image column is like: {“bytes”: , “path”: None}.
2. Creating a Dataset with `from_pandas`:

```auto
features = Features(
    {
        "image": DatasetImage(decode=False),
        "xml_content": Value("string"),
        "filename": Value("string"),
        "project_name": Value("string"),
    }
)

dataset = Dataset.from_pandas(df, preserve_index=False, features=features)

```

1. (Optional) Splitting it before make it iterable and iterate over train and test?
2. Flattening, create an `IterableDataset`, and go for all the mapping:

```auto
dataset = dataset.flatten_indices()
dataset = dataset.to_iterable_dataset()
dataset = dataset.map(...)

```

The mapping never puts the images encoded in the dataset.  
5. Persisting with `to_parquet()` - that’s it.

My problem is the caching/handling of the dataset before making it iterable.  
With a small dataset, it is very fast. But I also have some dataframes with over 15’000 rows.  
In those cases, the `from_pandas()` creates an OOM (also with 64 GB RAM).

Any idea and/or hint how to handle this?  
If I would save the images temp from the ZIP-file to disk and let the Dataset load the images - would that help?

Thanks in advance!

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [November 20, 2025, 2:09am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdatasets-caching-from-pandas%2F170648%2F2 "2025-11-20T02:09:54Z")

</div>

I don’t know [if this method actually works](https://huggingface.co/datasets/John6666/forum2/blob/main/dataset_caching_from_pandas_1.md), but…