# What is happening in the trainer api, with data collator?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhat-is-happening-in-the-trainer-api-with-data-collator%2F38139%3E%5C
**Category:** Beginners\
**Created:** [April 29, 2023, 7:04pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhat-is-happening-in-the-trainer-api-with-data-collator%2F38139 "2023-04-29T19:04:56Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![arnab9learns](https://avatars.discourse-cdn.com/v4/letter/a/57b2e6/32.png) [@arnab9learns](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Farnab9learns)%5C
**Post date:** [April 29, 2023, 7:04pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhat-is-happening-in-the-trainer-api-with-data-collator%2F38139%2F1 "2023-04-29T19:04:57Z")

</div>

class CustomDataset(Dataset):  
def **init** (self, evaluate: bool = False):

```
    self.examples = []
    self.sample_size=10 # change it to something meaningful later

    self.src_files = dpy

    for i in range(self.sample_size):
        sentences=dpy['train']['whole_func_string'][i].split('\n')
        self.examples+=[t.ids for t in tokenizer.encode_batch(sentences)]

def __len__ (self):
    return len(self.examples)

def __getitem__ (self, i):
    # We’ll pad at the batch level.
    return torch.tensor(self.examples[i], dtype=torch.int64)

```

d=CustromDataset()

# Define the Data Collator

data\_collator = DataCollatorForLanguageModeling(  
tokenizer=fast\_tokenizer, mlm=True, mlm\_probability=0.15, return\_tensors=‘pt’  
)  
from transformers import Trainer, TrainingArguments

training\_args = TrainingArguments(  
output\_dir=“./bert”,  
overwrite\_output\_dir=True,  
num\_train\_epochs=100,  
do\_train=True,  
per\_gpu\_train\_batch\_size=32,  
save\_steps=500,  
save\_total\_limit=2

)

trainer = Trainer(  
model=model,  
args=training\_args,  
train\_dataset=d,  
eval\_dataset=d,  
data\_collator=data\_collator  
)

Question: My question is here train\_dataset is d and also eval\_dataset is d, Does the data\_collator mask the train\_dataset and train against original eval\_dataset d? Or, Do, I need to provide completely new dataset here?  
what is the role the data\_collator playing here in the training, what does it exactly do?
