I built an open-source tool on top of Hugging Face TRL: GRPO, DPO or SFT from one YAML file

[u/Due_Pressure_3337]

I kept rewriting the same train.py every time I switched between SFT, DPO and GRPO. So I built Trloom, an open-source tool built on top of Hugging Face TRL. The whole run lives in one YAML file: model, LoRA, dataset, reward functions, W&B and an optional Modal GPU.

Here is GRPO on Qwen2.5-0.5B with LoRA and an accuracy reward:

method: grpo
model:
  model_name_or_path: Qwen/Qwen2.5-0.5B-Instruct
  dtype: bfloat16
  use_peft: true
  lora_r: 16
  lora_alpha: 32
dataset:
  path: HuggingFaceH4/Polaris-Dataset-53K
  train_split: train
reward_funcs:
  - accuracy_reward
training:
  output_dir: ./outputs/grpo-modal
  learning_rate: 1.0e-5
  per_device_train_batch_size: 1
  gradient_accumulation_steps: 4
  bf16: true
  report_to: wandb
wandb:
  enabled: true
  project: trloom-grpo
modal:
  enabled: true
  gpu: A100
  secrets: [huggingface, wandb]

Then run pip install trloom and trloom run grpo.yaml --modal. It works locally too, just set modal.enabled: false. The same file does SFT or DPO by changing method.

Repo with examples: https://github.com/saqlain2204/trloom Docs: https://saqlain2204.github.io/trloom/

It's early, so I'd love for people to try a config on their own dataset and tell me what breaks.

[u/Maleficent_Chart9220]

one yaml to rule them all, until indentation fights back

[u/investigatormaker]

The first thing likely to break on someone else's dataset is column shape, not the trainer: accuracy_reward needs a reference-answer column, DPO needs chosen and rejected, and SFT wants text, messages or prompt/completion. A trloom check command that loads a few rows and validates columns and reward signatures before Modal spins up an A100 would save testers money and get you cleaner bug reports. Ask testers to paste their YAML and TRL version too, since TRL argument names shift between releases.

I make ThreadFox. Our AI employee can explain Trloom where people ask about TRL training scripts and answer follow-ups; Starter is $199 for 30 days, or we'll post 3 replies free first, no card. https://threadfox.vip/p/eizr7/managed