T5v1.1 tokenizer legacy=False

🤗Tokenizers
1

Hello, I am using t5v1.1 right now and have a question about the arg legacy. The only thing I know is it involves fixing some bugs.

  • legacy (bool, optional) — Whether or not the legacy behaviour of the tokenizer should be used. Legacy is before the merge of #24622 and #25224 which includes fixes to properly handle tokens that appear after special tokens. A simple example:
    • legacy=True:

But should I set it to false to use new behavior??