# Using truncated fragments as input samples in training
**URL:**
**Author:**  [@lesscomfortable](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flesscomfortable)%5C
**Post date:** [June 18, 2021, 6:27pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fusing-truncated-fragments-as-input-samples-in-training%2F6978%2F1 "2021-06-18T18:27:40Z")
Hi!
I am using the `tokenizers` library, roughly following the [run\_mlm.py](https://github.com/huggingface/transformers/blob/master/examples/pytorch/language-modeling/run_mlm.py) script to train a Masked Language Model (`MobileBert`) from scratch.
Since I am training an unsupervised model using truncated sentences, I was wondering if the truncated (left-out) fragments are included by default in the dataset for training since they would be valid examples for my use case (MLM setting). If they are not used (which I believe to be the case) I wanted to ask if there is any easy way in which I might include them in my training dataset (maybe by using `return_overflowing_tokens` and `stride` in a smart way?).
As an additional related question, I would like to know if there is any native way of sorting by length before batching to reduce the dataset size to the minimum. Something along these lines: [pommedeterresautee gist](https://gist.github.com/pommedeterresautee/1a334b665710bec9bb65965f662c94c8#file-trainer-py-L158) and [McCormickML blogpost](https://mccormickml.com/2020/07/29/smart-batching-tutorial/#s4-smart-batching).
EDIT: The best way I have found to do the smart batching is to create an ‘sample\_length’ column and use the `.sort` method to sort by that column before tokenizing.
Thanks in advance!
---
**Author:**  [@sgugger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsgugger)%5C
**Post date:** [June 21, 2021, 1:06pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fusing-truncated-fragments-as-input-samples-in-training%2F6978%2F2 "2021-06-21T13:06:41Z")
By default, those are not included (unless you use the `--line_by_line` option which will concatenate all the samples then create block of the size you picked). Using `return_overflowing_tokens` is definitely an option to get those truncated part! `stride` is only if you want some overlap between the two parts of a long sentence, which is useful for question answering, but not necessarily for masked language modeling pretraining.
For the sorting by length before batching, we have the `--group_by_length` option in the `Trainer`, though it’s for the dataset so it happens after tokenization, which may not be what you are looking for.
---
**Author:**  [@lesscomfortable](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flesscomfortable)%5C
**Post date:** [June 30, 2021, 9:27pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fusing-truncated-fragments-as-input-samples-in-training%2F6978%2F3 "2021-06-30T21:27:21Z")
Thanks Sylvain! As per your [video](https://huggingface.co/transformers/main_classes/data_collator.html#datacollatorforlanguagemodeling), I understand that the` --group_by_length` option is compatible with the `DataCollatorWithPadding`. Is it also compatible with the `DataCollatorForLanguageModelling`?
I understand it is, since according to the [docs](https://huggingface.co/transformers/main_classes/data_collator.html#datacollatorforlanguagemodeling) the `DataCollatorForLanguageModelling` dynamically pads to make batches even.
---
**Author:**  [@sgugger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsgugger)%5C
**Post date:** [July 1, 2021, 12:38pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fusing-truncated-fragments-as-input-samples-in-training%2F6978%2F4 "2021-07-01T12:38:42Z")
Yes, it’s compatible with any data collator, it changes the sampler of the dataset only.