# How do you tokenize one long string?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-do-you-tokenize-one-long-string%2F44370%3E%5C
**Category:** Beginners\
**Created:** [June 24, 2023, 4:53am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-do-you-tokenize-one-long-string%2F44370 "2023-06-24T04:53:11Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![DiggingLace](https://avatars.discourse-cdn.com/v4/letter/d/90ced4/32.png) [@DiggingLace](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FDiggingLace)%5C
**Post date:** [June 24, 2023, 4:53am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-do-you-tokenize-one-long-string%2F44370%2F1 "2023-06-24T04:53:11Z")

</div>

Let’s say my training dataset is just one super long string. What is the correct way to tokenize this?

I have this so far:

`trainenc = tokenizer(train_dataset['text'], return_tensors='pt', max_length=128, truncation=True, padding=True, return_overflowing_tokens=True)`

What arguments should i keep? Afterwards, how do I split up my long list of tokens into batches where each element of the batch is short enough to fit inside the model?

Thanks
