# Trainer Ignoring Weight Decay, Beta arguments

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrainer-ignoring-weight-decay-beta-arguments%2F48541%3E%5C
**Category:** Beginners\
**Created:** [July 28, 2023, 4:38pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrainer-ignoring-weight-decay-beta-arguments%2F48541 "2023-07-28T16:38:13Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![slyle](https://avatars.discourse-cdn.com/v4/letter/s/e36b37/32.png) [@slyle](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fslyle)%5C
**Post date:** [July 28, 2023, 4:38pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrainer-ignoring-weight-decay-beta-arguments%2F48541%2F1 "2023-07-28T16:38:13Z")

</div>

Hello, this is probably an easy one to answer.

Trainer seems to be ignoring my weight decay and adam\_beta arguments. I will be training a model for many epochs that consist of many steps, and I want to slow down the rate at which the learning rate falls off so I don’t end up with virtually 0 learn rate after a few epochs.

Specifying weight\_decay=0, and increasing adam\_beta1 and adam\_beta2 do not seem to do anything to the magnitude of learning rate decay.

Am I missing something here? Or is this intended behavior? The weight decay also seems to be altered on a by step basis. I don’t see any arguments that allow me to change the “weight decay strategy”. Is the only way to alter that by subclassing Trainer and overriding the scheduler?

---

<div class="post-metadata">

**Author:** ![slyle](https://avatars.discourse-cdn.com/v4/letter/s/e36b37/32.png) [@slyle](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fslyle)%5C
**Post date:** [July 28, 2023, 9:45pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrainer-ignoring-weight-decay-beta-arguments%2F48541%2F2 "2023-07-28T21:45:14Z")

</div>

It turns out it wasn’t ignoring my arguments, but that I was trying to solve the wrong problem.

I needed to leave the Adam arguments alone. Even with 0 weight decay, there is a learn rate scheduler that defaults to a linear rate decay. You need to override the Trainer Class and create a custom scheduler as seen in other forum posts, and you can adjust power to be lower for slower decrease in LR:

```auto
class CustomTrainer(Trainer):
    def __init__ (self, *args, **kwargs):
        super(). __init__ (*args, **kwargs)

    def create_optimizer_and_scheduler(self, num_training_steps):
        self.optimizer = torch.optim.AdamW(self.model.parameters(),
                               lr=self.args.learning_rate,
                               weight_decay=self.args.weight_decay)
        self.lr_scheduler = get_polynomial_decay_schedule_with_warmup(
            self.optimizer, 0, num_training_steps, power=0.5)
 
trainer = CustomTrainer(
   model=model,
   args=training_args,
   train_dataset=input_ds['train'],
   eval_dataset=input_ds['test'],
   tokenizer=tokenizer,
   compute_metrics=compute_metrics
)

```