# How to convert Tokenizer to TokenizerFast?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-convert-tokenizer-to-tokenizerfast%2F1305%3E%5C
**Category:** Beginners\
**Created:** [September 29, 2020, 9:32pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-convert-tokenizer-to-tokenizerfast%2F1305 "2020-09-29T21:32:51Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![brianbargh](https://avatars.discourse-cdn.com/v4/letter/b/e480ec/32.png) [@brianbargh](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fbrianbargh)%5C
**Post date:** [September 29, 2020, 9:32pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-convert-tokenizer-to-tokenizerfast%2F1305%2F1 "2020-09-29T21:32:51Z")

</div>

I’m trying to run the question answering pipeline on chunks of of a long document. The DistilBertTokenizerFast has good features to support this. It can break a document down into overlapping chunks and it can give you the offsets mapping so that you can match up labels with tokens. But in the pipeline they use by default the DistilBertTokenizer, which has neither of these features. Is there a better way than below to find out what model is being used in the pipeline and construct a “fast” tokenizer to match?

```auto
from transformers import pipeline
from transformers import DistilBertTokenizerFast

nlp = pipeline("question-answering")
regular_tokenizer = nlp.tokenizer
# look up which model is being used on github 
# https://github.com/huggingface/transformers/blob/master/src/transformers/pipelines.py
# currently the model is:
model_name = "distilbert-base-cased-distilled-squad"
fast_tokenizer = DistilBertTokenizerFast.from_pretrained(model_name)

```

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvalhalla)%5C
**Post date:** [September 30, 2020, 2:57pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-convert-tokenizer-to-tokenizerfast%2F1305%2F2 "2020-09-30T14:57:27Z")

</div>

You can pass `tokenizer` to the `pipeline`

`pipeline("", tokenizer=your_tok)`