# Should I use .map(processor) or define tokenizer=processor?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fshould-i-use-map-processor-or-define-tokenizer-processor%2F61313%3E%5C
**Category:** 🤗Transformers\
**Created:** [November 7, 2023, 3:15am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fshould-i-use-map-processor-or-define-tokenizer-processor%2F61313 "2023-11-07T03:15:14Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Diegulio](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/diegulio/32/20973_2.png) [@Diegulio](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FDiegulio)%5C
**Post date:** [November 7, 2023, 3:15am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fshould-i-use-map-processor-or-define-tokenizer-processor%2F61313%2F1 "2023-11-07T03:15:14Z")

</div>

I am working in a vision use case. I have the processor:

```auto
processor = ViTImageProcessor.from_pretrained('google/vit-base-patch16-224')

```

Then I process my dataset as follow:

```auto
def apply_processor(example):
  example['pixel_values'] = processor(example['image'].convert("RGB"), return_tensors="pt").pixel_values.squeeze()
  return example

processed_dataset = pet_dataset.map(apply_processor)

```

Considering this, should I **also** add the `tokenizer = processor` in the transformers.Trainer ? If not, which one is the best option, doing the map/transform/etc or doing the tokenizer=processor?

Thanks in advance!