# How can I make sure Tokenizer pads to a fixed length?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-can-i-make-sure-tokenizer-pads-to-a-fixed-length%2F16116%3E%5C
**Category:** 🤗Tokenizers\
**Created:** [March 26, 2022, 7:34pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-can-i-make-sure-tokenizer-pads-to-a-fixed-length%2F16116 "2022-03-26T19:34:41Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![ollibolli](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ollibolli/32/16121_2.png) [@ollibolli](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Follibolli)%5C
**Post date:** [March 26, 2022, 7:34pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-can-i-make-sure-tokenizer-pads-to-a-fixed-length%2F16116%2F1 "2022-03-26T19:34:41Z")

</div>

Hi, I’m trying to use Distilbert as a layer in keras, however the tokenizer doesn’t pad to a fixed length but rather just some minimum depending on the batch. I guess that is expected reading up on it.  
However that doesn’t work since the input layer (because I’m combining) needs a fixed length.  
Can I somehow make sure the Tokenizer always pads to max\_length ?

thanks for any help and insights 🙂

---

<div class="post-metadata">

**Author:** ![merve](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/merve/32/49809_2.png) [@merve](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmerve)%5C
**Post date:** [March 28, 2022, 11:28am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-can-i-make-sure-tokenizer-pads-to-a-fixed-length%2F16116%2F2 "2022-03-28T11:28:49Z")

</div>

Hello and welcome to our forum 🤗

When you’re passing your sequences, can you set padding to “max\_length” and pass a value to the max\_length argument like so:

```auto
tokenizer(sequence, return_tensors="tf", padding="max_length", max_length=15)

```

Let me know if it works 🙂

---

<div class="post-metadata">

**Author:** ![ollibolli](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ollibolli/32/16121_2.png) [@ollibolli](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Follibolli)%5C
**Post date:** [March 29, 2022, 3:02pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-can-i-make-sure-tokenizer-pads-to-a-fixed-length%2F16116%2F3 "2022-03-29T15:02:56Z")

</div>

Hi merve,  
thanks a lot, this is something I tried, and many others, it didn’t work.  
I tried with padding = True, max\_length = 512 (or just fixed length) as well.

Weirdly a deprecated command does pad\_to\_max\_length = True and the outputs are full length. Very strange. But I’m happy this works now. 🙂

I’m thinking the issue is maybe that none of the text in the dataset are very long, if I make it some small length it works with just the padding/max\_length args.

Edit: I’m also thinking, maybe this is wanted behaviour and I don’t quite understand it, lol. 🤗
