Post-Training Techniques and Training Platforms: A Technical Reference

LoRA · QLoRA · Full Fine-Tuning · DPO · GRPO across Axolotl, Oumi, LLaMA-Factory, Unsloth, TRL, torchtune, NVIDIA NeMo, LLM Foundry, Ludwig, and PEFT

TL;DR

Five techniques form a ladder of increasing cost and capability: LoRA (cheap adapter tuning) → QLoRA (LoRA on a 4-bit frozen base, lowest VRAM) → FFT (all weights, highest quality/highest cost) on the supervised axis; then DPO (offline preference alignment, cheap) and GRPO (online RL with verifiable rewards, by far the most expensive and finicky). For an animal-nutrition adapter where numbers must come from a solver, the right path is SFT/LoRA first, optionally DPO, and GRPO only as a targeted final stage to reward valid tool-calls/schema.
Platform support is near-universal for LoRA/QLoRA/FFT/DPO, but GRPO cleanly separates the field. TRL, Unsloth, Axolotl, LLaMA-Factory, Oumi (via TRL/verl), Ludwig and NVIDIA NeMo-RL support GRPO; torchtune’s development was wound down in 2025 (successor “Forge” is itself paused, consolidating into torchtitan), and LLM Foundry has no GRPO/DPO and only limited LoRA. Easiest-to-hardest per technique differs: LoRA is easiest on LLaMA-Factory/Axolotl; GRPO is most turnkey on Unsloth (single GPU) and TRL, hardest on NeMo-RL/verl.
The day-to-day burden scales with the technique, not just the platform: LoRA/QLoRA are a ~10-line YAML or ~30-line Python job that runs on one 16–24 GB GPU in an afternoon; DPO adds a reference model and preference-data plumbing; GRPO adds reward functions, a vLLM generation server, group sampling and RL instability, needs multiple GPUs, and is roughly 10–100× more expensive per unit of improvement.

Key Findings

LoRA (Hu et al., 2021, arXiv:2106.09685) freezes the base and learns a low-rank update ΔW = BA scaled by α/r. Per the paper’s abstract: “Compared to GPT-3 175B fine-tuned with Adam, LoRA can reduce the number of trainable parameters by 10,000 times and the GPU memory requirement by 3 times,” with zero inference latency after merging. It “learns less and forgets less” than FFT (Biderman et al., 2024) — a regularizer, not a full replacement.
QLoRA (Dettmers et al., 2023, arXiv:2305.14314) is “an efficient finetuning approach that reduces memory usage enough to finetune a 65B parameter model on a single 48GB GPU while preserving full 16-bit finetuning task performance,” via 4-bit NF4 + double quantization + paged optimizers — but it is slower per step than LoRA because of dequantization.
FFT trains all parameters; the ~16 bytes/param mixed-precision AdamW rule means an 8B model needs ~128 GB just for weights+grads+optimizer states before activations, forcing ZeRO/FSDP sharding.
DPO (Rafailov et al., 2023, arXiv:2305.18290) turns RLHF into a single classification-style loss over preference pairs, using the model as its own implicit reward model; β≈0.1 controls the KL constraint. Needs an SFT’d base first.
GRPO (Shao et al., DeepSeekMath 2024, arXiv:2402.03300) drops PPO’s critic by normalizing rewards within a group of G sampled completions; it shines with verifiable rewards (RLVR) — exactly the feed-formulation case where a solver can check answers.

Details

PART 1 — THEORY AND MECHANISM

LoRA (Low-Rank Adaptation)

Core claim & math. Hu et al. (2021, arXiv:2106.09685) hypothesize that the weight update during adaptation has a low “intrinsic rank,” inspired by Aghajanyan et al.’s finding that fine-tuning happens on a low-dimensional manifold. For a pretrained matrix W₀ ∈ ℝ^(d×k), they constrain the update: W₀ + ΔW = W₀ + BA, where B ∈ ℝ^(d×r), A ∈ ℝ^(r×k), r ≪ min(d,k). The forward pass becomes h = W₀x + (α/r)·BAx. W₀ is frozen; only A and B receive gradients.

Initialization. A is random Gaussian, B is zero, so ΔW = BA = 0 at the start of training — the model begins from exactly the pretrained behavior and diverges gradually, which is far more stable than a random perturbation. The α/r scaling means you don’t need to retune the learning rate when you change r; the paper sets α to the first r tried and largely leaves it.

Which modules / empirical guidance. The original paper adapted only attention projections (W_q, W_v), finding rank as low as r=1–4 competitive. Modern practice (and QLoRA) targets all linear layers — q_proj, k_proj, v_proj, o_proj plus the MLP gate_proj, up_proj, down_proj — which is what target_modules="all-linear" does in PEFT and gives performance closer to full fine-tuning. Typical starting points: r = 8–32 (up to 64–128 for harder domain shift), α = r or 2r, dropout 0.05 (0.0 for small models per QLoRA).

Memory math. LoRA’s savings come overwhelmingly from the optimizer state, not the weights. Mixed-precision AdamW stores, per trainable parameter, ~2 bytes gradient + 4 bytes fp32 master weight + 4+4 bytes for Adam’s m and v — the optimizer states dominate. Because LoRA makes <1% of parameters trainable, that 12+ bytes/param overhead applies to a tiny slice. The frozen base still occupies memory (16-bit) but carries no optimizer state. Per Hu et al.’s abstract, this yields a 10,000× reduction in trainable parameters and a 3× reduction in GPU memory vs Adam full fine-tuning of GPT-3 175B.

Trainable %. For an 8B model, rank-16 LoRA on attention+MLP is on the order of 0.1–1% of parameters trainable (e.g., ~131k params per adapted matrix vs ~16.8M full, a 128× reduction at r=16 for a 4096×4096 matrix).

Merging vs serving. After training, B and A can be merged into W₀ (merge_and_unload()), giving zero inference latency — indistinguishable from the base at serve time. Alternatively you keep adapters separate and swap/serve many dynamically (e.g., S-LoRA, vLLM multi-LoRA), trading a small runtime overhead for the ability to host many domain adapters on one base. Note the practical warning that merged weights can occasionally drift format adherence — validate post-merge.

Variants & whether they’re worth it.

rsLoRA (rank-stabilized): scales by α/√r instead of α/r, preventing gradient collapse at high rank; worth it if you use r ≥ 64. In PEFT: use_rslora=True.
DoRA (weight-decomposed, Liu et al. 2024): splits weight into magnitude + direction, applies LoRA to direction; closes part of the FFT gap at small extra cost. use_dora=True. (QDoRA with bitsandbytes can be flaky — issues reported on DeepSpeed Zero2/ 8-bit.)
LoRA+: higher LR for B than A; faster convergence; via PEFT’s LoraPlus optimizer, not a config flag.
PiSSA / LoftQ: smarter initialization (principal singular values / quantization-aware) that speeds early training and reduces quantization error respectively — useful for QLoRA.
ReLoRA: periodically merges and restarts to accumulate higher effective rank during pretraining-scale runs.

Verdict: rsLoRA and DoRA are the two most broadly worthwhile; the rest are situational.

When LoRA underperforms FFT. Biderman et al., “LoRA Learns Less and Forgets Less” (arXiv:2405.09673, TMLR 2024), compared LoRA and full fine-tuning on programming and mathematics in both the instruction-finetuning regime (≈100K prompt-response pairs) and continued pretraining (≈20B unstructured tokens). In standard low-rank settings LoRA substantially underperforms full fine-tuning, but forgets less of the base model’s out-of-domain ability (it’s a stronger regularizer than weight decay/dropout). The paper reports that “full finetuning learns perturbations with a rank that is 10-100× greater than typical LoRA configurations, possibly explaining some of the reported gaps.” Implication for domain adaptation: LoRA is excellent for style/format/behavior adaptation and preserving general ability; for injecting substantial new knowledge, a new language/vocabulary, or continued pretraining, FFT (or high-rank LoRA + more data) wins. Follow-up analyses note the conclusion is learning-rate sensitive — LoRA typically needs a higher LR than FFT.

QLoRA

Paper & three innovations. Dettmers et al. (2023, arXiv:2305.14314) present “an efficient finetuning approach that reduces memory usage enough to finetune a 65B parameter model on a single 48GB GPU while preserving full 16-bit finetuning task performance”; their best model family, Guanaco, reaches “99.3% of the performance level of ChatGPT while only requiring 24 hours of finetuning on a single GPU.” The three innovations are: (a) 4-bit NormalFloat (NF4), a data type information-theoretically optimal for zero-centered normally distributed weights, built on quantile quantization so each bin holds an equal expected mass of a normal distribution; (b) Double Quantization, “a method that quantizes the quantization constants, saving an average of about 0.37 bits per parameter (approximately 3 GB for a 65B model)”; (c) paged optimizers, using NVIDIA unified memory to page optimizer state to CPU during gradient-checkpointing memory spikes, preventing OOM.

Mechanism. The base model is stored frozen in 4-bit NF4; LoRA adapters are kept in bf16. During forward/backward, 4-bit weights are dequantized on the fly to bf16 for the matmul, then discarded; gradients flow only into the bf16 adapters. NF4 + double quantization matches bf16; the paper standardizes on r=64, α=16, LoRA on all linear layers, bf16 compute.

Memory numbers (rules of thumb). 7B/8B QLoRA fits in ~6–12 GB (runs on a 16 GB card comfortably); 13B ~10–16 GB; 65B/70B fits in ~46–48 GB on a single A100/L40S (the paper’s headline result); 405B needs multi-GPU (roughly ~2 × 80 GB+ even in 4-bit just for weights). torchtune reports its 8B QLoRA recipe peaks below ~10 GB vs ~19 GB for plain LoRA.

Speed/quality tradeoff. QLoRA is slower per step than plain LoRA — typically noticeably slower — because of the dequantization overhead on every forward/backward; you trade wall-clock speed for a large VRAM reduction. Quality is very close to LoRA/16-bit for most instruction-tuning workloads.

bitsandbytes dependency. QLoRA in the HF stack depends on bitsandbytes (NF4 kernels, paged optimizers, 8-bit Adam). This is a hard CUDA dependency that has historically been the #1 source of environment breakage on non-standard platforms (older CUDA, Windows, some AKS base images). Alternatives/competitors: HQQ (fast half-quadratic quantization), AWQ/GPTQ-based 4-bit training paths, torchao NF4 (PyTorch-native, what torchtune uses), and FP8/NVFP4 QAT on Hopper/Blackwell for quantization-aware training.

Full Fine-Tuning (FFT)

Definition. All parameters trainable.

Memory math in detail. The standard mixed-precision AdamW accounting is ~16 bytes per parameter: 2 bytes bf16 weight + 2 bytes bf16 gradient + 4 bytes fp32 master weight + 4 bytes Adam m + 4 bytes Adam v = 16 bytes. For an 8B model that’s ~128 GB before activations — already beyond a single 80 GB GPU. Activations add more and scale with batch × sequence length.

Why distributed is forced. You shard with DeepSpeed ZeRO or PyTorch FSDP/FSDP2:

ZeRO-1: shard optimizer states across GPUs.
ZeRO-2: + shard gradients.
ZeRO-3: + shard parameters (equivalent to FSDP full-shard) — each GPU holds a slice of everything and gathers on demand.

For very large models you add tensor parallelism (split matmuls), pipeline parallelism (split layers), and context/sequence parallelism (split long sequences).

Activation memory & checkpointing. Gradient (activation) checkpointing recomputes activations in the backward pass instead of storing them — trading ~30% extra compute for large activation-memory savings. Nearly always on for FFT.

Memory-reducing optimizers. 8-bit Adam (bitsandbytes) halves optimizer state; Adafactor and Lion store less state; GaLore projects gradients to low rank; LOMO fuses backward+update to avoid storing gradients; Muon (newer) is a momentum-orthogonalizing optimizer gaining traction.

When FFT is worth it. Large domain shift, new knowledge injection, new language/vocabulary, continued pretraining, or when you need the absolute quality ceiling and have the compute. Otherwise LoRA/QLoRA usually suffices and is safer for retention.

Catastrophic forgetting & mitigation. FFT forgets more (Biderman et al.). Mitigate with replay/data mixing (blend in general-domain data), lower learning rate, fewer epochs, and evaluating out-of-domain deltas (e.g., MMLU) as a guardrail.

DPO (Direct Preference Optimization)

Paper & theory. Rafailov et al. (2023, arXiv:2305.18290), “Your Language Model Is Secretly a Reward Model.” Starting from the KL-constrained RLHF objective under the Bradley-Terry preference model, they show the optimal policy has a closed form, and reparameterize the reward in terms of the policy itself. This lets you solve RLHF as a single classification loss on preference data — no explicit reward model, no sampling loop, no RL. The paper reports DPO matches or exceeds PPO-based RLHF while “being substantially simpler to implement and train.”

Loss, term by term. L_DPO = −log σ( β·[ log(π_θ(y_w|x)/π_ref(y_w|x)) − log(π_θ(y_l|x)/π_ref(y_l|x)) ] ) where y_w is the chosen (winning) response, y_l the rejected. The bracket is the difference of implicit rewards (log-ratio of trained policy to frozen reference) for chosen vs rejected. σ is the logistic function; maximizing the log-sigmoid pushes the chosen implicit reward above the rejected by a margin scaled by β.

Reference model & the LoRA trick. π_ref is a frozen copy of the SFT model. Holding two models doubles memory — but with LoRA/PEFT you can compute the reference logits by simply disabling the adapter on the same base weights, so you don’t need a second model in memory. TRL and most wrappers do this automatically for PEFT DPO.

β hyperparameter. β controls the strength of the KL constraint to the reference. Standard is β = 0.1. Too low → the policy drifts far from the reference, instability, reward hacking, degeneration. Too high → over-regularized, barely moves from SFT (terse/evasive outputs); fix by lowering β or mixing in fresh SFT steps.

Data & prerequisites. Format is (prompt, chosen, rejected) triples. Typically thousands to ~100k pairs. SFT must come first — DPO on a non-SFT’d base is unstable because the implicit reward is anchored to the reference.

Failure modes. Likelihood displacement — the probability of the chosen response can actually decrease during training (both chosen and rejected log-probs fall, just rejected faster); watch that reward margins rise while chosen logps don’t collapse. Also verbosity/length bias, reward hacking, and distribution shift when preference data is off-policy.

Variant family.

IPO: adds a regularizer to avoid DPO overfitting to deterministic preferences (loss_type="ipo").
KTO (Kahneman-Tversky): uses unpaired binary good/bad labels — ideal when you only have thumbs-up/down, not pairs.
ORPO: merges SFT + preference into one stage with no reference model (odds-ratio penalty) — one-stage, memory-light.
SimPO: reference-free, length-normalized implicit reward.
CPO, RPO, APO: other reference-free / robustness variants.
DPO with an explicit length penalty to fight verbosity.

GRPO (Group Relative Policy Optimization)

Paper & prominence. Shao et al., DeepSeekMath (2024, arXiv:2402.03300); made famous by DeepSeek-R1 (arXiv:2501.12948).

Theory. PPO needs a value/critic network (often as large as the policy) to estimate advantages. GRPO removes it: for each prompt, sample a group of G completions from the old policy, score each with a reward r_i, and compute the advantage as the group-normalized reward: Â_i = (r_i − mean({r})) / std({r}) Every token in completion i gets that same outcome advantage. This halves the model memory vs PPO (no critic to train/hold) and works well even with sparse rewards because normalization preserves gradient signal.

Objective. GRPO optimizes a PPO-style clipped surrogate: J = E[ (1/G) Σ_i (1/|o_i|) Σ_t { min(ρ_{i,t}·Â_{i,t}, clip(ρ_{i,t}, 1−ε, 1+ε)·Â_{i,t}) − β·D_KL } ] where ρ_{i,t} is the token-level probability ratio π_θ/π_θ_old, ε the clip range, and β the KL penalty against a frozen reference. Unlike PPO’s GAE (which needs the critic), advantages come purely from group statistics.

RLVR — why it matters here. GRPO shines when you have a programmatic, verifiable reward — math correctness, code passing tests, format/schema compliance, constraint satisfaction — rather than a learned reward model. This “RL with verifiable rewards” is exactly the feed-formulation setting: a solver can verify whether the model emitted a valid tool call with correct ingredient IDs and whether the solver accepted the constraint set. The reward is objective and cheap to compute, which is GRPO’s ideal regime.

Reward functions in practice. In TRL a reward function has signature def reward(completions, **kwargs) -> list[float] (Axolotl: def my_reward(completions, **kwargs) -> list[float], with completions[i][0]["content"] = text). You can compose multiple reward functions (e.g., one for format/schema, one for solver acceptance) with per-function weights (reward_weights).

Key hyperparameters. num_generations (group size G, commonly 8; DeepSeek used up to 32–64), max_completion_length, beta (KL coef; DeepSeek-R1-Zero and DAPO actually drop KL, i.e. β=0), temperature, num_iterations, learning rate ~1e-6 to 5e-6.

Computational cost. Generation dominates — you produce G completions per prompt every step. This makes vLLM integration essentially mandatory; TRL/Axolotl/Unsloth all wire in vLLM in either colocate mode (vLLM shares training GPUs) or server mode (dedicated GPUs). GRPO is dramatically more expensive and finicky than DPO or SFT — expect roughly an order of magnitude (or more) more wall-clock per unit of improvement, plus RL instability (reward collapse, entropy collapse, length blowup).

Known biases & successors.

Dr. GRPO (“Understanding R1-Zero-like Training,” arXiv:2503.20783): the original per-response length normalization and std-normalization introduce a length/difficulty bias; Dr. GRPO divides by a constant (max completion length) instead. In TRL: loss_type="dr_grpo".
DAPO (Yu et al., ByteDance, arXiv:2503.14476): four fixes — decoupled (“clip-higher”) bounds, dynamic sampling (drop zero-advantage groups), token-level gradient loss, overlong reward shaping; drops KL. Per the paper it “achieves 50 points on AIME 2024 based on Qwen2.5-32B model, outperforming previous state-of-the-art results achieved by DeepSeek-R1-Zero-Qwen-32B (47 points) using 50% training steps” — while their initial vanilla GRPO run “reached only 30 points on AIME.”
GSPO (Group Sequence Policy Optimization, Zheng et al., Qwen Team/Alibaba, arXiv:2507.18071): “defines the importance ratio based on sequence likelihood and performs sequence-level clipping, rewarding, and optimization,” which “notably stabilizes Mixture-of-Experts (MoE) RL training”; the paper states “these merits of GSPO have contributed to the remarkable improvements in the latest Qwen3 models.”

TRL implements GRPO with selectable loss_type (incl. dr_grpo) and much of DAPO’s toolkit; Axolotl exposes GRPO with importance-sampling controls (importance_sampling_level: token, vllm_importance_sampling_correction); Unsloth supports GRPO/GSPO-style runs.

How to tell if each is working.

LoRA/QLoRA/FFT (SFT): watch training/eval loss decreasing smoothly; watch for overfitting (eval loss turns up), and check an out-of-domain benchmark delta for forgetting.
DPO: watch reward margin (chosen − rejected implicit reward) rising and reward accuracy (fraction where chosen > rejected) climbing above 0.5 toward ~0.7–0.9; watch that chosen logps don’t collapse (likelihood displacement).
GRPO: watch mean reward trending up, completion length (should stabilize, not blow up or collapse), and KL to reference (should stay bounded — spikes mean the policy is drifting/hacking).

Typical 8B recipe starting points.

LoRA SFT: LR 1e-4 to 2e-4, effective batch 8–64, 1–3 epochs, cosine scheduler, warmup 3–10%, r=16–32. ~afternoon on one 24 GB GPU; datasets from ~1k (style) to ~100k (broad) examples.
QLoRA SFT: same but LR often 2e-4, paged_adamw_8bit optimizer; runs on 16 GB.
FFT: LR 5e-6 to 2e-5, effective batch large, 1–3 epochs; needs 8× 80 GB (ZeRO-3/FSDP) for 8B.
DPO: LR 5e-6 (5e-7 to 1e-5), β=0.1, 1–3 epochs, effective batch 16–64, cosine + 10% warmup; thousands to ~100k pairs; single 24–80 GB GPU with LoRA.
GRPO: LR 1e-6 to 5e-6, G=8, β 0–0.04, 1 epoch over prompts, max_completion_length 512–1024; needs vLLM + ideally ≥2 GPUs; hundreds to thousands of steps.

PART 2 — CONFIGURATION AND WORKFLOW PER PLATFORM

1. Axolotl (YAML-driven)

Axolotl wraps Transformers/PEFT/TRL/DeepSpeed behind one YAML. CLI: axolotl preprocess config.yaml, axolotl train config.yaml, axolotl merge-lora config.yaml, plus axolotl vllm-serve for GRPO. (There’s also axolotl agent-docs {sft,grpo,preference_tuning,...} and axolotl config-schema for the current key list.)

LoRA / QLoRA / FFT (toggle by adapter + quantization):

base_model: meta-llama/Meta-Llama-3.1-8B-Instruct
# LoRA: load_in_8bit: true, adapter: lora
# QLoRA:
load_in_4bit: true
adapter: qlora          # lora | qlora | (blank/omit for full FT)
lora_r: 32
lora_alpha: 16
lora_dropout: 0.05
lora_target_linear: true      # target all linear layers
# lora_target_modules: [q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj]
peft_use_dora: true           # optional DoRA
peft_use_rslora: true         # optional rsLoRA
datasets:
  - path: yahma/alpaca-cleaned
    type: alpaca
sequence_len: 2048
sample_packing: true
micro_batch_size: 2
gradient_accumulation_steps: 4
learning_rate: 0.0002
num_epochs: 3
optimizer: paged_adamw_8bit
lr_scheduler: cosine
bf16: auto
gradient_checkpointing: true
flash_attention: true

For full fine-tune: omit adapter, set load_in_4bit/8bit: false, add deepspeed: deepspeed_zero3.json (required for 8B+).

DPO (and ORPO/KTO/SimPO/IPO via the rl: key):

base_model: meta-llama/Meta-Llama-3.1-8B-Instruct
rl: dpo                    # dpo | ipo | kto | simpo | orpo | grpo
rl_beta: 0.1               # (also expressible as trl.beta)
adapter: lora
lora_r: 16
lora_alpha: 32
lora_target_linear: true
datasets:
  - path: argilla/ultrafeedback-binarized-preferences
    type: chat_template.default    # or chatml.intel for the classic intel format
    field_chosen: chosen
    field_rejected: rejected
learning_rate: 5e-6

With LoRA, Axolotl uses the disabled-adapter base as the implicit reference — no second model.

GRPO (needs rl: grpo, a reward module, and vLLM). Two-terminal server mode:

base_model: Qwen/Qwen2.5-1.5B-Instruct
vllm:
  host: 0.0.0.0
  port: 8000
  gpu_memory_utilization: 0.85
  dtype: auto
  max_model_len: 4096
rl: grpo
trl:
  use_vllm: true
  vllm_server_host: 0.0.0.0
  vllm_server_port: 8000
  beta: 0.001
  num_generations: 8
  max_completion_length: 512
  reward_funcs: [rewards.accuracy_reward]   # "{file}.{fn}" importable from cwd
  # reward_weights: [1.0]
datasets:
  - path: AI-MO/NuminaMath-TIR
    type: rewards.prompt_transform
learning_rate: 1e-5

# Terminal 1 (dedicated GPU for generation — must be the LAST GPUs due to TRL/vLLM):
CUDA_VISIBLE_DEVICES=2,3 axolotl vllm-serve grpo.yaml
# Terminal 2 (training):
CUDA_VISIBLE_DEVICES=0,1 axolotl train grpo.yaml --num-processes 2

Reward signature: def my_reward(completions, **kwargs) -> list[float]. Axolotl adds async prefetch (async_prefetch), LoRA weight-sync (vllm_lora_sync), streaming scoring (streaming_partial_batch), and importance-sampling controls (vllm_importance_sampling_correction, importance_sampling_level: token). GRPO landed in v0.7.0 and is marked “Beta.” Known sharp edges: GRPO+QLoRA+DeepSpeed-Z3 adapter merges have reported bugs; reward-function path errors have historically failed silently.

2. Oumi (YAML-driven, CLI: `oumi train/evaluate/infer/launch`)

Oumi is a config-schema wrapper with model:, data:, training:, peft:, fsdp: blocks. It routes RL to TRL or ByteDance’s verl.

SFT/LoRA/QLoRA: training.trainer_type: TRL_SFT, peft.lora_r, peft.q_lora: true.
DPO: trainer_type: TRL_DPO.
GRPO: trainer_type: TRL_GRPO or VERL_GRPO; set grpo.use_vllm: true, specify reward_functions (registered via Oumi’s registry / @register_dataset) and optionally a rollout_function. The TRL path shares HF hyperparameters and Oumi’s GrpoParams maps onto TRL’s GRPOConfig; the verl path “exposes much more hyperparameters (200+ total), and natively supports GRPO for vision-language models” but requires Ray. Launch locally with oumi train -c config.yaml or remotely with oumi launch.

Oumi’s value is uniform configs + easy remote job launching; the trade-off is you inherit whichever backend’s quirks (verl = powerful but unfamiliar/Ray-heavy).

3. LLaMA-Factory (YAML + LlamaBoard Gradio WebUI; CLI `llamafactory-cli train/chat/export/webui`)

Stage-based: stage: sft | dpo | kto | ppo | rm | pt; finetuning_type: lora | full | freeze; quantization_bit: 4 for QLoRA.

LoRA/QLoRA SFT:

model_name_or_path: Qwen/Qwen3-4B-Instruct-2507
stage: sft
do_train: true
finetuning_type: lora
quantization_bit: 4          # omit for plain LoRA; finetuning_type: full for FFT
lora_rank: 8
lora_alpha: 16
lora_dropout: 0.05
lora_target: all
dataset: identity,alpaca_en_demo
template: qwen3
cutoff_len: 2048
per_device_train_batch_size: 1
gradient_accumulation_steps: 8
learning_rate: 1.0e-4
num_train_epochs: 3.0
lr_scheduler_type: cosine
warmup_ratio: 0.1
bf16: true
output_dir: saves/qwen3-4b/lora/sft

llamafactory-cli train config.yaml (override inline: ... learning_rate=1e-5).

DPO / preference (sigmoid=DPO, plus ORPO, SimPO):

stage: dpo
finetuning_type: lora
lora_target: all
pref_beta: 0.1
pref_loss: sigmoid          # sigmoid (DPO) | hinge | ipo | orpo | simpo
dataset: dpo_en_demo
template: llama3
learning_rate: 5.0e-6

KTO uses stage: kto. PPO uses stage: ppo with a reward_model: path. Multi-node FFT/DPO via FORCE_TORCHRUN=1 NNODES=... llamafactory-cli train .... GRPO is not a native stage — LLaMA-Factory’s RLHF is SFT/RM/PPO/DPO/KTO-centric; for GRPO you use TRL/Axolotl/Unsloth. LlamaBoard GUI (llamafactory-cli webui) lets you pick model/dataset/method, set LoRA/quant knobs, launch, and watch loss curves — the lowest-friction on-ramp for non-scripters.

4. Unsloth (Python/notebook API; single-GPU speed/VRAM optimizer)

from unsloth import FastLanguageModel
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/Meta-Llama-3.1-8B-bnb-4bit",
    max_seq_length=2048,
    load_in_4bit=True,          # True = QLoRA; False = 16-bit LoRA
    fast_inference=True,        # enable vLLM (for GRPO)
    max_lora_rank=32,
    gpu_memory_utilization=0.6,
)
model = FastLanguageModel.get_peft_model(
    model, r=32,
    target_modules=["q_proj","k_proj","v_proj","o_proj",
                    "gate_proj","up_proj","down_proj"],
    lora_alpha=32,              # often r or 2r
    lora_dropout=0,             # 0 is optimized in Unsloth
    bias="none",
    use_gradient_checkpointing="unsloth",   # ~30% less VRAM
    random_state=3407,
    use_rslora=False,           # rsLoRA supported
    loftq_config=None,          # LoftQ supported
)

Then hand model to TRL’s SFTTrainer, DPOTrainer (call PatchDPOTrainer() first in older versions), or GRPOTrainer. GRPO on Unsloth is the most turnkey single-GPU RL experience: fast_inference=True colocates vLLM, and GRPOConfig(use_vllm=True, num_generations=8, ...) runs on one consumer/L40S-class GPU. Unsloth is Python-first (no YAML), fastest, lowest VRAM (its docs claim ~2–5× faster and ~50–70% less VRAM than naive HF+bitsandbytes), but single-GPU-centric (multi-GPU is newer/limited) and its custom kernels occasionally break on bleeding-edge model architectures.

5. HuggingFace TRL (library + `trl` CLI)

The substrate most others wrap. Python: SFTTrainer/SFTConfig, DPOTrainer/DPOConfig, GRPOTrainer/GRPOConfig, all accepting a peft_config=LoraConfig(...) and a quantization_config=BitsAndBytesConfig(...) for QLoRA. CLI supports trl sft|dpo|grpo|kto|reward|rloo, plus trl vllm-serve.

trl sft --model_name_or_path Qwen/Qwen2.5-0.5B --dataset_name stanfordnlp/imdb
trl dpo --config dpo_config.yaml

from trl import GRPOConfig, GRPOTrainer
def reward_len(completions, **kwargs):     # reward signature
    return [ -abs(50 - len(c)) for c in completions ]
cfg = GRPOConfig(use_vllm=True, vllm_mode="colocate",  # or "server"
                 num_generations=8, max_completion_length=256,
                 beta=0.0, loss_type="dr_grpo")
trainer = GRPOTrainer(model="Qwen/Qwen2.5-0.5B", reward_funcs=[reward_len],
                      args=cfg, train_dataset=ds)
trainer.train()

vLLM: colocate (shares training GPUs, good for single-GPU/Colab) or server (trl vllm-serve on dedicated GPUs, better throughput). DPO auto-handles the PEFT implicit reference. QLoRA GRPO/DPO by combining BitsAndBytesConfig + peft_config. TRL is the most current (fastest to get new methods: IPO/KTO/SimPO/ORPO, dr_grpo, GSPO-style), most flexible, and the reference implementation — but you write Python and wire pieces yourself. Scale with accelerate launch --config_file deepspeed_zero3.yaml.

6. torchtune (recipes + YAML) — maintenance wound down

Status: Meta/PyTorch halted active torchtune development in 2025 (GitHub issue #2883, “The future of torchtune”). It received only critical bug/security fixes through 2025; last release ~v0.6.1; PyPI/Snyk flag it “Inactive/discontinued.” The team said they were building “a new product in a new repo … a simple native PyTorch solution for end-to-end post-training with scale as a first-class citizen.” That successor is meta-pytorch/torchforge (“Forge”), open-sourced ~Oct 2025, built on Monarch + vLLM + TorchTitan + TorchStore, and RL-first (SFT and GRPO). Forge is explicitly experimental (“early development. Expect bugs, incomplete features, and API changes”), and its README now states “Development in Forge has paused. LLM training at PyTorch is being consolidated in torchtitan.” Practical guidance: do not start new production work on torchtune or Forge; use TRL/Axolotl/Unsloth. torchtune’s recipes remain a good reference implementation for learning.

For completeness, torchtune’s workflow (still runnable on pinned versions): tune ls (list recipes/configs), tune cp (copy a config to edit), tune download, tune run <recipe> --config <config>. It shipped 21 recipes spanning SFT, knowledge distillation, DPO, PPO, GRPO, and QAT; recipes include lora_finetune_single_device, lora_finetune_distributed, full_finetune_single_device, full_finetune_distributed, lora_dpo_single_device, plus QLoRA configs (e.g. llama3_1/8B_qlora_single_device). Configs use _component_: instantiation (dotted path to a Python class/factory). Examples:

tune run lora_finetune_single_device --config llama3_1/8B_lora_single_device
tune run lora_finetune_single_device --config llama3_1/8B_qlora_single_device
tune run --nproc_per_node 2 lora_finetune_distributed --config llama3/8B_lora
tune run lora_dpo_single_device --config llama2/7B_lora_dpo_single_device

8B QLoRA peaks <10 GB; LoRA ~19 GB.

7. NVIDIA NeMo (NeMo 2.0 + NeMo-RL + NeMo Customizer)

Three distinct products:

(a) NeMo 2.0 Framework — Python/NeMo-Run config. PEFT is a callback, not a YAML flag:

from nemo.collections import llm
recipe = llm.llama3_8b.finetune_recipe(
    name="llama3_8b_ft", dir="/checkpoints",
    num_nodes=1, num_gpus_per_node=8,
    peft_scheme="lora",     # "lora" | "dora" | "none" (=full SFT)
    packed_sequence=False,
)
recipe.peft.dim = 16        # LoRA rank
recipe.peft.alpha = 32
recipe.peft.target_modules = ["linear_qkv","linear_proj","linear_fc1","linear_fc2"]

Or via the fine-tune API: llm.finetune(..., peft=llm.peft.LoRA(target_modules=['linear_qkv','linear_proj'], dim=32)). NeMo-Run CLI registers lora/dora/none as factories: peft=lora. Note NeMo 2.0 renamed LoRA targets from NeMo 1.0’s ['attention_qkv','attention_dense','mlp_fc1','mlp_fc2'] to ['linear_qkv','linear_proj','linear_fc1','linear_fc2']. There is also NeMo AutoModel (Hugging Face-native, Triton LoRA kernels, _target_: nemo_automodel...PeftConfig with dim/alpha/use_triton).

(b) NeMo-RL (NVIDIA-NeMo/RL) — DPO, GRPO, RM at scale with Megatron or DTensor-V2 backends and vLLM generation. LoRA for SFT landed in v0.5, extended to GRPO and DPO in v0.6:

policy:
  dtensor_cfg:
    lora_cfg:
      enabled: true
      dim: 128
      alpha: 512
      match_all_linear: true

(Megatron backend uses policy.megatron_cfg.peft with dim/alpha/exclude_modules.) GRPO recipe example: examples/configs/recipes/llm/grpo-llama3.2-1b-instruct-...megatron_generation.yaml. This is the NVIDIA path for GRPO/DPO on multi-node clusters.

(c) NeMo Customizer (enterprise microservice) — supports SFT-LoRA, Full SFT (all_weights), DPO, and Knowledge Distillation (GRPO is not here — that’s NeMo-RL). You POST a customization config (target model + training_options) and a job (dataset + hyperparameters):

from nemo_microservices import NeMoMicroservices
client = NeMoMicroservices(base_url=os.environ['CUSTOMIZER_BASE_URL'])
config = client.customization.configs.create(
    name="llama-3.1-8b-instruct@v1.0.0+80GB",
    namespace="default",
    target="meta/llama-3.1-8b-instruct@2.0",
    training_options=[{"training_type":"sft","finetuning_type":"lora",
                       "num_gpus":2,"tensor_parallel_size":1}],
    training_precision="bf16", max_seq_length=2048,
)

LoRA job body:

{ "hyperparameters": {
    "training_type":"sft","finetuning_type":"lora",
    "epochs":10,"batch_size":16,"learning_rate":0.0001,
    "lora": { "adapter_dim": 8, "adapter_dropout": 0.1 } } }

training_type ∈ {sft, dpo}, finetuning_type ∈ {lora, all_weights}. Datasets are namespaced (default/my-dataset, JSONL prompt/completion). Setting lora_enabled=True in the deployment config auto-creates a NIM deployment for the base model and serves the trained LoRA adapter — the enterprise inference path. This is the most “managed” but most opinionated/heavyweight option, and the right fit if you’re already on NVIDIA AI Enterprise / NIM.

8. MosaicML LLM Foundry (Databricks)

Composer/FSDP-based, YAML configs run via composer train/train.py <yaml>. Its strength is large-scale FFT and pretraining; LoRA is a secondary, PEFT-wrapped feature. LoRA (add peft_config to the model block):

model:
  name: hf_causal_lm
  pretrained: true
  pretrained_model_name_or_path: mistralai/Mistral-7B-v0.1
  peft_config:
    r: 16
    peft_type: LORA
    task_type: CAUSAL_LM
    lora_alpha: 32
    lora_dropout: 0.05
    target_modules:
      - q_proj
      - k_proj

LoRA support is via the PEFT integration and is comparatively limited/less-exercised (for MPT-style models it approximates the fused Wqkv). Full fine-tune is the primary path (hf_causal_lm + FSDP config + streaming dataloader). No DPO/GRPO in core LLM Foundry — preference/RL is out of scope; this maps to Databricks Mosaic AI Model Training for managed runs (llm-foundry[gpu], e.g. 0.20.0; FSDP, MLflow, Unity Catalog checkpoints, serverless GPU). Choose LLM Foundry only if you’re a Databricks shop doing large SFT/pretraining; it’s the wrong tool for an adapter-first or RL workflow.

9. Ludwig (declarative YAML)

Ludwig (v0.17) advertises SFT, DPO, KTO, ORPO, GRPO (new), and a large PEFT menu (LoRA, DoRA, VeRA, LoRA+, PiSSA, etc.) with 4-bit QLoRA via torchao.

model_type: llm
base_model: meta-llama/Meta-Llama-3.1-8B
adapter:
  type: lora
  r: 16
  alpha: 32
  dropout: 0.05
quantization:
  bits: 4          # QLoRA
trainer:
  type: finetune
  learning_rate: 0.0002
  epochs: 3
input_features: [{name: prompt, type: text}]
output_features: [{name: response, type: text}]

Maintenance caveat: Ludwig’s docs and site advertise the full feature set including GRPO (v0.17), but the project’s release cadence has slowed markedly and it is the least actively developed of the mainstream options here — verify the current release before committing, and treat GRPO support as newer/less battle-tested than TRL’s. Ludwig’s real differentiator is being a multi-modal declarative framework (tabular/text/image/audio) with an HPO + serving stack, not an LLM-RL specialist.

10. HuggingFace PEFT (the substrate)

PEFT is the library Axolotl, TRL, LLaMA-Factory, Unsloth and LLM Foundry all call for LoRA/QLoRA. Core API:

from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training

bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
                         bnb_4bit_use_double_quant=True,
                         bnb_4bit_compute_dtype="bfloat16")     # QLoRA
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf",
                                             quantization_config=bnb, device_map="auto")
model = prepare_model_for_kbit_training(model)   # grad-checkpointing, cast norms
cfg = LoraConfig(r=16, lora_alpha=32, target_modules="all-linear",
                 lora_dropout=0.05, bias="none", task_type="CAUSAL_LM",
                 use_rslora=False, use_dora=False)   # flags for rsLoRA/DoRA
model = get_peft_model(model, cfg)
model.print_trainable_parameters()

target_modules="all-linear" = the QLoRA all-linear recipe. rsLoRA (use_rslora=True, α/√r scaling), DoRA (use_dora=True), LoftQ (replace_lora_weights_loftq / init_lora_weights="loftq") are config flags/utilities. Merge with model.merge_and_unload(). PEFT is where you go when a wrapper hides a knob you need. (Reference syntax verified against PEFT v0.17.)

PART 3 — COMPARATIVE WORKFLOW EFFORT

Support matrix (✅ native/easy · 🟡 partial/via-backend/harder · ❌ not supported):

Technique	Axolotl	Oumi	LLaMA-Factory	Unsloth	TRL	torchtune*	NeMo	LLM Foundry	Ludwig	PEFT
LoRA	✅	✅	✅	✅	✅	✅	✅	🟡	✅	✅
QLoRA	✅	✅	✅	✅	✅	✅	🟡	🟡	✅	✅
FFT	✅	✅	✅	🟡	✅	✅	✅	✅	✅	n/a
DPO	✅	✅	✅	✅	✅	✅	✅ (NeMo-RL/Customizer)	❌	✅	n/a
GRPO	✅	✅ (TRL/verl)	❌	✅	✅	🟡 (legacy)	✅ (NeMo-RL)	❌	🟡 (new)	n/a

*torchtune development wound down 2025 — capable but unmaintained; successor “Forge” is paused.

Time-to-first-successful-run (competent engineer, fresh start):

LoRA/QLoRA: LLaMA-Factory (WebUI) or Axolotl ~15–30 min; Unsloth notebook ~15 min; TRL/PEFT ~30–60 min (write script); torchtune ~20 min (copy config); NeMo Framework ~1–3 h (heavier setup); LLM Foundry ~1–2 h.
DPO: Axolotl/LLaMA-Factory ~30–45 min (need preference data formatted); TRL ~1 h; NeMo-RL/Customizer hours.
GRPO: Unsloth notebook ~1 h (single GPU, reward fns provided); TRL ~2–4 h (reward fns + vLLM); Axolotl ~3–6 h (two-terminal vLLM, reward module, path gotchas); NeMo-RL/verl ~1–2 days (cluster, Ray).

Config vs Python: Axolotl/LLaMA-Factory/Oumi/Ludwig = pure YAML (~10–40 lines). torchtune = YAML + occasional recipe edit. TRL/PEFT = Python (~20–60 lines). Unsloth = Python notebook (~30–50 lines). NeMo 2.0 = Python config objects; Customizer = REST/SDK JSON. LLM Foundry = YAML + Composer.

What breaks most / debug difficulty:

bitsandbytes/CUDA mismatches (all QLoRA paths) — most common single failure, especially on custom AKS images.
Chat-template / loss-masking mistakes (training on prompt tokens) — silent quality killer; Axolotl (train_on_inputs: false), LLaMA-Factory (template:), TRL (completion-only collator) mostly handle it.
DPO: forgetting SFT-first; wrong preference field mapping; reference-model OOM without PEFT.
GRPO: vLLM wiring (GPU placement — TRL/Axolotl require vLLM on the last GPUs), reward-function import path (Axolotl historically failed silently), reward collapse / length blowup, OOM from group generation.

Does the platform handle the annoying parts automatically?

Chat template & completion-only loss: Axolotl, LLaMA-Factory, Unsloth, TRL — yes; raw PEFT — you do it.
Packing: Axolotl (sample_packing), LLaMA-Factory, NeMo (packed_sequence) — yes.
DPO reference handling: TRL/Axolotl/LLaMA-Factory auto-use disabled-adapter base with PEFT — yes.
vLLM wiring for GRPO: Unsloth (colocate, automatic) easiest; TRL (colocate/server flags); Axolotl (explicit two-terminal); NeMo-RL (backend-managed).
Adapter merging: Axolotl (axolotl merge-lora), LLaMA-Factory (export), PEFT (merge_and_unload), Unsloth (save_pretrained_merged).

Easiest vs hardest per technique:

LoRA/QLoRA: easiest = LLaMA-Factory (WebUI) / Unsloth; hardest = LLM Foundry.
FFT: easiest = Axolotl/LLaMA-Factory (bundled DeepSpeed/FSDP configs) or LLM Foundry at scale; hardest = raw TRL+accelerate (you wire ZeRO yourself).
DPO: easiest = LLaMA-Factory / Axolotl (one key); hardest = NeMo.
GRPO: easiest = Unsloth (single GPU) then TRL; hardest = verl (via Oumi) / NeMo-RL (cluster complexity).

Operational burden / upgrade fragility: TRL and Unsloth move fast — configs/APIs can break on minor version bumps (pin versions). Axolotl and LLaMA-Factory are relatively stable YAML surfaces but track TRL underneath. torchtune/Forge = maintenance risk. NeMo = heavy containers (nvcr.io/nvidia/nemo:*) but very stable once pinned. GRPO runs need the most babysitting (watch reward/KL/length live); SFT/LoRA runs are largely fire-and-forget.

PART 4 — GUIDANCE FOR THE ANIMAL-NUTRITION / FEED-FORMULATION ADAPTER

The key architectural fact: numeric answers must come from a solver/tool, not from the weights. So you are not trying to teach the model arithmetic or to memorize nutrient tables — you’re teaching it to (1) speak the domain, (2) map user intent to the right canonical ingredient IDs and constraints, and (3) emit a valid, correctly-structured tool call that your solver can consume, then narrate the solver’s result faithfully. That reframing determines which techniques matter.

Recommended order:

SFT with LoRA (or QLoRA) first — this is 80% of the value. Build a dataset of (user request → correct tool call with canonical ingredient IDs + constraint schema → grounded natural-language answer). LoRA is the right tool: you’re adapting behavior and format, not injecting large new factual knowledge, and Biderman et al. show LoRA both suffices for this and forgets less of the base model’s general ability — valuable because you want the model to stay a competent generalist that happens to speak feed-formulation. Use r=16–32, α=32, all-linear targets, LR 1e-4–2e-4, 1–3 epochs. QLoRA if you’re VRAM-constrained on AKS (fits an 8B on a single 16–24 GB GPU); plain LoRA if you have a 40–80 GB card and want speed. Merge the adapter for serving, or serve it dynamically via vLLM multi-LoRA on AKS.
DPO only if you have preference signal on style/faithfulness. If SFT leaves residual issues — the model sometimes hallucinates numbers instead of calling the solver, or is verbose/hedgy — collect (chosen, rejected) pairs where “chosen” calls the tool and reports faithfully and “rejected” fabricates or rambles. β=0.1, LR 5e-6, LoRA so the reference is free. DPO is cheap and stabilizes behavior, but it can’t guarantee schema validity — it only nudges probabilities.
GRPO as a targeted final stage — genuinely promising here, precisely because your reward is verifiable. This is the strongest fit for RLVR of any use case: your reward function is not a fuzzy learned model, it’s your actual solver. Compose rewards, e.g.:

+1 if the emitted tool call parses as valid JSON/schema (format reward);
+1 if every ingredient ID is in the canonical set (grounding reward);
+1 if the solver accepts the constraint set as feasible / returns a valid formulation (execution reward);
small shaped penalties for extra prose or invalid IDs.

Weight them (reward_weights) and let GRPO push the policy toward completions that your solver actually accepts. This directly optimizes the metric you care about — “did the model produce something the solver can run correctly” — which SFT (imitation) and DPO (pairwise preference) only optimize indirectly. Use loss_type="dr_grpo" to avoid the length bias.

What GRPO costs vs the SFT+LoRA path — be clear-eyed:

SFT+LoRA: one 16–80 GB GPU, an afternoon per iteration, cheap to babysit, deterministic, easy to debug. Gets you a working tool-calling feed-formulation assistant.
GRPO: needs generation infrastructure (vLLM colocate or a dedicated GPU server), realistically ≥2 GPUs (one for training, one for generation) even for an 8B, group sampling (G=8) that makes each step ~8× the generation cost, RL instability to monitor (reward/KL/length curves), and roughly 10–100× the wall-clock and engineering effort of the SFT run for the incremental gain. You also must make the solver callable inside the reward loop with low latency, or generation stalls.

Verdict for the user: Do SFT+LoRA first and ship it — measure tool-call validity rate and solver-acceptance rate as your north-star metrics. If those plateau below your bar (say valid-schema rate stuck at 85–90% and you need 99%+), then add a GRPO stage with the solver-as-reward — it is one of the few production settings where GRPO’s cost is clearly justified, because the reward is free, exact, and exactly the business metric. Skip GRPO entirely if SFT+DPO already clears your validity threshold. Platform-wise: prototype SFT/LoRA and DPO in Axolotl or LLaMA-Factory (YAML, fast iteration) or Unsloth (if single-GPU on AKS); do the GRPO stage in TRL (most current, dr_grpo loss, cleanest reward-function API) or Unsloth (if constrained to one GPU), with vLLM for both training-time generation and production serving on AKS. Reserve NeMo-RL/Customizer for if you standardize on NVIDIA AI Enterprise/NIM and need multi-node scale.

Recommendations

Start now with QLoRA SFT on an 8B instruct model in LLaMA-Factory or Axolotl: r=16–32, α=32, all-linear, LR 2e-4, 3 epochs, train_on_inputs:false. Dataset = (request → tool call → grounded answer). Target metric: tool-call schema-validity and solver-acceptance rate, not loss. Threshold to advance: if validity ≥ your bar, ship; if not, continue.
If behavior issues remain, add a LoRA DPO pass (β=0.1, LR 5e-6) on (faithful-tool-call vs fabricated) pairs. Cheap, low-risk.
Only if validity/acceptance still misses the bar, run a GRPO stage with the solver as a composed verifiable reward (format + canonical-ID + solver-acceptance), in TRL (or Unsloth if single-GPU), with vLLM and loss_type="dr_grpo". Budget ≥2 GPUs and real monitoring. Change trigger: adopt GRPO when the gap between “imitatable” and “verifiable-correct” behavior is what’s blocking you.
Serving on Azure/AKS: use vLLM with multi-LoRA to host the merged/adapter model; the same vLLM stack doubles as GRPO’s generation backend, so standardizing on it early pays off twice.
Pin your stack. bitsandbytes/CUDA and TRL/Unsloth versions are the top upgrade-breakage sources; containerize (or use NeMo’s pinned images) and freeze versions per training run.
Avoid torchtune/Forge and LLM Foundry for this project — torchtune is unmaintained, Forge is paused/experimental, and LLM Foundry lacks DPO/GRPO and only partially supports LoRA.

Caveats

Config keys drift between versions. Examples reflect 2025–2026 docs (Axolotl rl:/trl: blocks and GRPO v0.7.0 “Beta”, TRL loss_type="dr_grpo", LLaMA-Factory pref_loss, NeMo 2.0 renamed LoRA targets, PEFT v0.17 use_dora/use_rslora). Verify against the exact pinned version — TRL renames args across minor releases, and NeMo 1.0→2.0 changed the whole PEFT surface.
torchtune status: development wound down in 2025 (issue #2883); successor “Forge” (meta-pytorch/torchforge) is experimental and its README now says development is paused and LLM training at PyTorch is consolidating into torchtitan. Treat both as reference-only.
Ludwig maintenance is uncertain; its site claims GRPO/DPO/full PEFT menu (v0.17) but cadence has slowed — verify before committing and treat GRPO as newer/less-tested than TRL’s.
LLM Foundry LoRA is a limited PEFT integration; no native DPO/GRPO. Its managed sibling is Databricks Mosaic AI Model Training.
GRPO cost figures are order-of-magnitude, not benchmarked on your hardware; actual cost depends heavily on completion length, G, and solver latency inside the reward loop.
Memory figures (16 bytes/param; QLoRA fits an 8B in ~6–12 GB; ~128 GB for 8B FFT; 65B QLoRA on 48 GB is the QLoRA paper’s headline) are standard rules of thumb / paper claims; real peak memory varies with batch, sequence length, activation checkpointing, and parallelism.
Some snippets are drawn from vendor tutorials and third-party guides corroborating official docs; where a platform’s own docs were thin (Ludwig GRPO, Oumi GRPO schema specifics) the uncertainty is flagged rather than over-asserted.

Part of the Univrs research ecosystem:

Research: metalearn.org · ardeshir.io · sepahsalar.org · univrs.io
Creative: imagine.univrs.io
Code: github.com/univrs · github.com/ardeshir