TRL documentation

GMPO

You are viewing v1.18.0 version. A newer version v1.18.1 is available.
Hugging Face's logo
Join the Hugging Face community

and get access to the augmented documentation experience

to get started

GMPO

在论文 Geometric-Mean Policy Optimization 中,作者提出了一种 GRPO 变体,该变体最大化 token 级重要性比值的几何均值,而非算术均值。由于几何均值对异常比值远不敏感,策略更新更加稳定,并且能够容忍更宽的裁剪范围。裁剪在 log 空间中按 token 应用,并根据优势值的符号进行单侧裁剪(即标准的 PPO 信任域)——关键在于,这一操作发生在计算几何均值之前。

要使用 GMPO,您可以使用 trl.experimental.gmpo 中的 experimental.gmpo.GMPOTrainer 类。

用法

from trl.experimental.gmpo import GMPOConfig, GMPOTrainer

training_args = GMPOConfig(
    epsilon=0.4,  # log-space clip range -> ratios clipped to (exp(-0.4), exp(0.4)); paper, Sec. 4
    beta=0.0,
)
trainer = GMPOTrainer(
    model="Qwen/Qwen3-0.6B",
    reward_funcs=...,
    train_dataset=...,
    args=training_args,
)
trainer.train()

在 GMPO 中,裁剪操作应用于逐 token 的 log 重要性比率(即在 log 空间中进行),然后才计算几何平均值,因此 epsilon 和 epsilon_high 是在 log 空间中表达的:有效的比率裁剪范围为 (exp(-epsilon), exp(epsilon_high))。该论文建议采用比 GRPO/DAPO 明显更宽的范围 (exp(-0.4), exp(0.4)),以鼓励探索。

GMPOTrainer

类 trl.experimental.gmpo.GMPOTrainer

< >

( modelreward_funcsargs = None**kwargs )

用于几何平均策略优化(GMPO)的 Trainer。

GMPO(https://huggingface.co/papers/2507.20673)是 GRPO 的一种变体,它最大化 token 级重要性比值的几何均值,而非算术均值。由于几何均值对异常比值的敏感度远低于算术均值,策略更新更加稳定,并且可以使用更宽的截断范围。

与 GRPOTrainer 相比,唯一的区别在于 _compute_loss。其余所有内容(生成、奖励计算、权重同步、指标记录)均保持不变并直接继承。

训练

< >

( resume_from_checkpoint: str | bool | None = Nonetrial: optuna.Trial | dict[str, Any] | None = Noneignore_keys_for_eval: list[str] | None = None ) → ~trainer_utils.TrainOutput

参数

  • resume_from_checkpoint (str 或 bool,可选) — 如果为 str,则为之前 Trainer 实例保存的检查点的本地路径。如果为 bool 且等于 True,则加载之前 Trainer 实例保存在 args.output_dir 中的最后一个检查点。如果存在,训练将从此处加载的模型/优化器/调度器状态中恢复。
  • trial (optuna.Trial 或 dict[str, Any],可选) — 用于超参数搜索的试验运行或超参数字典。
  • ignore_keys_for_eval (list[str],可选) — 在训练期间收集预测结果以进行评估时,应忽略的模型输出(如果输出是字典)中的键列表。

返回值

~trainer_utils.TrainOutput

包含全局步数、训练损失和指标的 Object。

主要训练入口。

save_model

< >

( output_dir: str | None = None_internal_call: bool = False )

将保存模型,以便您可以使用 from_pretrained() 重新加载它。

仅从主进程保存。

push_to_hub

< >

( commit_message: str | None = 'End of training'blocking: bool = Truetoken: str | None = Nonerevision: str | None = Noneprivate: bool | None = None**kwargs )

参数

  • commit_message (str,可选,默认为 "End of training") — 推送时提交的提交信息。
  • blocking (bool,可选,默认为 True) — 是否仅在 git push 完成后才返回。
  • token (str,可选,默认为 None) — 具有写入权限的 Token,用于覆盖 Trainer 的原始参数。
  • revision (str, 可选) — 要从中提交内容的 git 版本。默认为 “main” 分支的头部。
  • private (bool,可选) — 创建仓库时是否将其设为私有。默认值为 args.hub_private_repo。如果仓库已存在,则此参数无效。
  • kwargs (dict[str, Any],可选) — 传递给 ~Trainer.create_model_card 的额外关键字参数。

将 self.model 和 self.processing_class 上传到 🤗 模型库中的仓库 self.args.hub_model_id。

GMPOConfig

类 trl.experimental.gmpo.GMPOConfig

< >

( output_dir: str | None = Noneper_device_train_batch_size: int = 8num_train_epochs: float = 3.0max_steps: int = -1learning_rate: float = 1e-06lr_scheduler_type: transformers.trainer_utils.SchedulerType | str = 'linear'lr_scheduler_kwargs: dict | str | None = Nonewarmup_steps: float = 0optim: transformers.training_args.OptimizerNames | str = 'adamw_torch_fused'optim_args: str | None = Noneweight_decay: float = 0.0adam_beta1: float = 0.9adam_beta2: float = 0.999adam_epsilon: float = 1e-08optim_target_modules: None | str | list[str] = Nonegradient_accumulation_steps: int = 1average_tokens_across_devices: bool = Truemax_grad_norm: float = 1.0label_smoothing_factor: float = 0.0bf16: bool | None = Nonefp16: bool = Falsebf16_full_eval: bool = Falsefp16_full_eval: bool = Falsetf32: bool | None = Noneactivation_checkpointing: bool | None = Noneactivation_checkpointing_kwargs: dict[str, typing.Any] | str | None = Nonegradient_checkpointing: bool = Truegradient_checkpointing_kwargs: dict[str, typing.Any] | str | None = Nonetorch_compile: bool = Falsetorch_compile_backend: str | None = Nonetorch_compile_mode: str | None = Noneuse_liger_kernel: bool = Falseliger_kernel_config: dict[str, bool] | None = Noneuse_cache: bool = Falseneftune_noise_alpha: float | None = Nonetorch_empty_cache_steps: int | None = Noneauto_find_batch_size: bool = Falselogging_strategy: transformers.trainer_utils.IntervalStrategy | str = 'steps'logging_steps: float = 10logging_first_step: bool = Falselog_on_each_node: bool = Truelogging_nan_inf_filter: bool = Trueinclude_num_input_tokens_seen: str | bool = 'no'log_level: str = 'passive'log_level_replica: str = 'warning'disable_tqdm: bool | None = Nonereport_to: None | str | list[str] = 'none'run_name: str | None = Noneproject: str = 'huggingface'trackio_space_id: str | None = Nonetrackio_bucket_id: str | None = Nonetrackio_static_space_id: typing.Union[str, NoneType, typing.Literal[False]] = Noneeval_strategy: transformers.trainer_utils.IntervalStrategy | str = 'no'eval_steps: float | None = Noneeval_delay: float = 0per_device_eval_batch_size: int = 8prediction_loss_only: bool = Falseeval_on_start: bool = Falseeval_do_concat_batches: bool = Trueeval_use_gather_object: bool = Falseeval_accumulation_steps: int | None = Noneinclude_for_metrics: list[str] = <factory>batch_eval_metrics: bool = Falsesave_only_model: bool = Falsesave_strategy: transformers.trainer_utils.SaveStrategy | str = 'steps'save_steps: float = 500save_on_each_node: bool = Falsesave_total_limit: int | None = Noneenable_jit_checkpoint: bool = Falsepush_to_hub: bool = Falsehub_token: str | None = Nonehub_private_repo: bool | None = Nonehub_model_id: str | None = Nonehub_strategy: transformers.trainer_utils.HubStrategy | str = 'every_save'hub_always_push: bool = Falsehub_revision: str | None = Noneload_best_model_at_end: bool = Falsemetric_for_best_model: str | None = Nonegreater_is_better: bool | None = Noneignore_data_skip: bool = Falserestore_callback_states_from_checkpoint: bool = Falsefull_determinism: bool = Falseseed: int = 42data_seed: int | None = Noneuse_cpu: bool = Falseaccelerator_config: dict | str | None = Noneparallelism_config: accelerate.parallelism_config.ParallelismConfig | None = Nonedataloader_drop_last: bool = Falsedataloader_num_workers: int = 0dataloader_pin_memory: bool = Truedataloader_persistent_workers: bool = Falsedataloader_prefetch_factor: int | None = Nonedataloader_multiprocessing_context: str | None = Nonedataloader_in_order: bool = Trueremove_unused_columns: bool | None = Falselabel_names: list[str] | None = Nonetrain_sampling_strategy: str = 'random'length_column_name: str = 'length'ddp_find_unused_parameters: bool | None = Noneddp_bucket_cap_mb: int | None = Noneddp_broadcast_buffers: bool | None = Noneddp_static_graph: bool | None = Noneddp_backend: str | None = Noneddp_timeout: int = 1800fsdp: str | None = Nonefsdp_config: dict[str, typing.Any] | str | None = Nonedeepspeed: dict | str | None = Nonedebug: str | list[transformers.debug_utils.DebugOption] = ''skip_memory_metrics: bool = Truedo_train: bool = Falsedo_eval: bool = Falsedo_predict: bool = Falseresume_from_checkpoint: str | None = Nonelocal_rank: int = -1model_init_kwargs: dict[str, typing.Any] | str | None = Nonetrust_remote_code: bool = Falserouter_aux_loss_coef: float | None = Nonedisable_dropout: bool = Falsecast_lm_head_to_fp32: bool = Falsenum_generations: int | None = 8num_generations_eval: int | None = Nonemax_completion_length: int | None = 512ds3_gather_for_generation: bool = Trueshuffle_dataset: bool | None = Truepad_to_multiple_of: int | str | None = 'auto'generation_batch_size: int | None = Nonesteps_per_generation: int | None = Nonetemperature: float = 1.0top_p: float = 1.0top_k: int = 0min_p: float | None = Nonegeneration_kwargs: dict | str | None = Nonechat_template_kwargs: dict | str | None = Nonerepetition_penalty: float = 1.0cache_implementation: str | None = Noneuse_vllm: bool = Falsevllm_mode: str = 'colocate'vllm_model_impl: str = 'vllm'vllm_enable_sleep_mode: bool = Falsevllm_structured_outputs_regex: str | None = Nonevllm_server_base_url: str | None = Nonevllm_server_host: str = '0.0.0.0'vllm_server_port: int = 8000vllm_server_timeout: float = 240.0vllm_group_port: int = 51216vllm_gpu_memory_utilization: float = 0.3vllm_max_model_length: int | None = Nonevllm_tensor_parallel_size: int = 1beta: float = 0.0num_iterations: int = 1epsilon: float = 0.4delta: float | None = Noneepsilon_high: float | None = Nonesapo_temperature_neg: float = 1.05sapo_temperature_pos: float = 1.0vespo_k_pos: float = 2.0vespo_lambda_pos: float = 3.0vespo_k_neg: float = 3.0vespo_lambda_neg: float = 2.0importance_sampling_level: str = 'token'reward_weights: list[float] | None = Nonemulti_objective_aggregation: str = 'sum_then_normalize'scale_rewards: str = 'group'loss_type: str = 'dapo'mask_truncated_completions: bool = Falsesync_ref_model: bool = Falseref_model_mixup_alpha: float = 0.6ref_model_sync_steps: int = 512top_entropy_quantile: float = 1.0entropy_coef: float = 0.0use_adaptive_entropy: bool = Falseentropy_coef_min: float = 0.0entropy_coef_max: float = 1.0entropy_coef_delta: float = 0.005entropy_target: float = 0.2max_tool_calling_iterations: int | None = Nonevllm_importance_sampling_correction: bool = Truevllm_importance_sampling_mode: str = 'sequence_mask'vllm_importance_sampling_clip_max: float | None = 3.0vllm_importance_sampling_clip_min: float | None = Noneoff_policy_mask_threshold: float | None = Noneuse_bias_correction_kl: bool = Truelog_completions: bool = Falselog_multimodal: bool = Truenum_completions_to_print: int | None = Nonelog_unique_prompts: bool = Falselog_completions_hub_repo: str | None = Noneuse_transformers_continuous_batching: bool = Falsetransformers_continuous_batching_config: dict | str | None = Noneuse_transformers_paged: bool = Falsevllm_importance_sampling_cap: float | None = None )

参数

  • epsilon (float, 可选, 默认为 0.4) — 下界裁剪值,以对数空间表示。每个 token 的重要性比率的下界为 exp(-epsilon)。
  • epsilon_high (float, 可选) — 上界裁剪值,以对数空间表示。如果为 None,则默认为 epsilon 的值。每个 token 的重要性比率的上界为 exp(epsilon_high)。

experimental.gmpo.GMPOTrainer 的配置类。

experimental.gmpo.GMPOConfig 继承了 GRPOConfig 的所有参数;它仅更改了裁剪范围的含义和默认值。在 GMPO 中,裁剪应用于每个 token 的 log 重要性比率(即在 log 空间中),然后才计算几何平均值,因此 epsilon 和 epsilon_high 以 log 空间表示:有效的比率裁剪范围为 (exp(-epsilon), exp(epsilon_high))。GMPO 论文 建议采用比 GRPO/DAPO 明显更宽的范围 (exp(-0.4), exp(0.4)),以鼓励探索。

Update on GitHub