# Correct formatting of Multi-Features Time Series dataset
**URL:**
**Author:**  [@admarcosai](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fadmarcosai)%5C
**Post date:** [July 17, 2023, 11:04am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcorrect-formatting-of-multi-features-time-series-dataset%2F47065%2F1 "2023-07-17T11:04:10Z")
Hello Everyone,
I am working on a Time Series Forecasting task with a .csv file containing in the order of millions of rows. I am trying to use the [TimeSeriesForecasting notebook](https://huggingface.co/blog/time-series-transformers) to establish a baseline. I am trying to use the following [datasets notebook](https://github.com/huggingface/notebooks/blob/main/examples/time_series_datasets.ipynb) to put my dataframe in the appropriate format for a huggingface Time Series dataset.
**The following is approwimately the structure of my dataframe:**
- index (COHORT\_MONTH): the index of the datarame is the month of the year so “freq=M”
- MSISDN (item\_id): int64 ====\> unique phone number identifier
- TOT\_DATA\_REV (target): float32 ====\> the target variable: real dynamic feature
- TOT\_OUTG\_REV: float32 ====\> independent variable: real dynamic feature
- TOT\_VOICE\_REV: float32 ====\> independent variable: real dynamic feature
- NB\_OUTG\_OFFNET\_SMS: int16 ====\> independent variable: dynamic feature
- NB\_OUTG\_INTER\_SMS: int16 ====\> independent variable: dynamic feature
The dataframe actually has in reality 31 dynamic features in total.
**Since all of the time series start at the same month and some have missing time steps,** I used the following code from [GluonTS pandasdataframe Tutorial](https://ts.gluon.ai/stable/tutorials/data_manipulation/pandasdataframes.html)
```auto
max_end = max(df.groupby("MSISDN").apply(lambda _df: _df.index[-1]))
dfs_dict = {}
for item_id, gdf in df.groupby("MSISDN"):
new_index = pd.date_range(gdf.index[0], end=max_end, freq="1D")
dfs_dict[item_id] = gdf.reindex(new_index).drop("MSIDN", axis=1)
ds = PandasDataset(dfs_dict, target="target")
```
My confusion comes from this code snippet in the hugging face tutorial:
```auto
from datasets import Dataset, Features, Value, Sequence
features = Features(
{
"start": Value("timestamp[s]"),
"target": Sequence(Value("float32")),
"feat_static_cat": Sequence(Value("uint64")),
# "feat_static_real": Sequence(Value("float32")),
# "feat_dynamic_real": Sequence(Sequence(Value("uint64"))),
# "feat_dynamic_cat": Sequence(Sequence(Value("uint64"))),
"item_id": Value("string"),
}
)
```
_From my understanding my case would only be concerned with start, target, feat\_dynamic\_real._
**NOW ARE MY QUESTIONS 🤗**
1- do I understand correctly when I think I have to create a dictionary with keys: start: min(COHORT\_MONTH), target(TOT\_DATA\_REV), feat\_dynamic\_real: [\* 31 real dynamic features], item\_id: MSISDN
I tried doing the above and it is excruciatingly slow 10000 MSISDN/40min when I have about 5millions
2- could just creating a Feature for each of my features work?
---
**Author:**  [@lhoestq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flhoestq)%5C
**Post date:** [July 17, 2023, 1:35pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcorrect-formatting-of-multi-features-time-series-dataset%2F47065%2F2 "2023-07-17T13:35:32Z")
> 2- could just creating a Feature for each of my features work?
Yup that does the job and would fit the notebook better, otherwise you’d end up with a `Sequence(Sequence(Value("float32")))` (31 \* window\_length floating values)
---
**Author:**  [@admarcosai](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fadmarcosai)%5C
**Post date:** [July 18, 2023, 11:15am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcorrect-formatting-of-multi-features-time-series-dataset%2F47065%2F3 "2023-07-18T11:15:18Z")
Trying to create a separate feature for each column kept giving me errors. It seems I didn’t understand how Dataset.from\_list(dict\_list, features=features) works as it actually.
I ended up using:
```auto
from datasets import Dataset, Features, Value, Sequence
features = Features(
{
"start": Value("timestamp[s]"),
"target": Sequence(Value("float32")),
"item_id": Value("string"),
"feat_dynamic_real": Sequence(Sequence(Value("uint64"))),
})
Dataset.from_list(dict_list, features=features)
```
**A Question Following These**
Is there a way to create train and test set from the resulting hugginface dataset without doubly storing the dataset like:
train\_set: dataset(2022-01-31 to 2022-12-301
test\_set: dataset(2022-01-31 to 2023-04-30)
---
**Author:**  [@lhoestq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flhoestq)%5C
**Post date:** [July 18, 2023, 5:22pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcorrect-formatting-of-multi-features-time-series-dataset%2F47065%2F4 "2023-07-18T17:22:16Z")
You can use `dataset.select(...)` and pass the indices that you’d use for train or test
---
**Author:**  [@admarcosai](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fadmarcosai)%5C
**Post date:** [July 19, 2023, 3:24am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcorrect-formatting-of-multi-features-time-series-dataset%2F47065%2F5 "2023-07-19T03:24:32Z")
Actually in the Time Series Transformer tutorial I am following the test and train set contain the same samples; only the test samples have more time steps than the train samples. This question might be more appropruate for the Transformers category I think