Hugging Face Forums

Meanings of different brackets during tokenization

🤗Transformers

Harveenchadha December 10, 2021, 10:42pm 1

Hi,

What is the meaning of different brackets during tokenization ?

Like [ ], {} , ()

Topic		Replies	Views	Activity
Ġ token inserted by ByteLevelBPETokenizer 🤗Transformers	0	582	November 1, 2023
Is it possible to tokenize multiple text modalities? 🤗Transformers	1	468	September 1, 2022
Printing tokens array 🤗Tokenizers	0	137	April 12, 2024
How can I get a list of word segmentation results for non-English string? 🤗Transformers	14	71	November 6, 2025
What is point of tokenizer.json in MT0? 🤗Transformers	0	228	November 24, 2022