# How did the dataset manages long sentences?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-did-the-dataset-manages-long-sentences%2F14334%3E%5C
**Category:** 🤗Datasets\
**Created:** [February 2, 2022, 9:38am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-did-the-dataset-manages-long-sentences%2F14334 "2022-02-02T09:38:44Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![peune](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/peune/32/3395_2.png) [@peune](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpeune)%5C
**Post date:** [February 2, 2022, 9:38am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-did-the-dataset-manages-long-sentences%2F14334%2F1 "2022-02-02T09:38:44Z")

</div>

Hello,

Given the following code:

```auto
tokenizer = GPT2Tokenizer.from_pretrained(xxx)

def encode(batch):
  return tokenizer(batch['text'], padding="max_length", add_special_tokens=True, truncation=True, max_length=64)

dataset = load_dataset("text", data_files = ['text.txt'])
dataset = dataset['train']
dataset.set_transform(encode)

```

I believe that each line in ‘text.txt’ will be pass through the ‘encode’ function that will apply the tokenizer then pad or truncate the result to exactly 64 tokens.  
Does this means that if a line is 100 tokens long, the token number 65-100 will never be used in training?

If this is the case, then should I manually split long lines in ‘text.txt’ into multiple lines?

Thanks

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/64813_2.png) [@lhoestq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flhoestq)%5C
**Post date:** [February 15, 2022, 4:09pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-did-the-dataset-manages-long-sentences%2F14334%2F2 "2022-02-15T16:09:10Z")

</div>

Hi ! That’s correct. You can use `.map()` to do so, see [how to split long sentences](https://huggingface.co/docs/datasets/process.html#split-long-examples) for example
