GAC · Noise-Aware Adaptive Mixing for Hybrid SFT-RL Post-Training

TL;DR

Hybrid SFT–RL post-training uses a fixed mixing weight μ that ignores how relative signal noise evolves during training. GAC casts μ as an MSE-optimal gradient estimation problem, gives a closed-form solution, and estimates it online from three cheap coefficient-space proxies. Result: +3.8 pp over HPT across math / code / science / logic, 28% less KL drift, < 1% wall-time overhead, and gains scale with model size (1.5B → 14B).

GAC treats the SFT/RL mixing weight μ as an estimation target with a closed-form MSE-optimal solution. Three coefficient-space proxies (σ_s², σ_r², Δg̃²) feed a guarded update stack (EMA → cosine-prior blend → per-step cap → clipping).

Abstract

Hybrid post-training that combines supervised fine-tuning (SFT) and on-policy reinforcement learning (RL) is the standard recipe for aligning large language models. Yet the mixing weight μ that balances the two signals is almost always a fixed, hand-tuned schedule. This is fragile: when RL noise spikes, or when the SFT expert starts pulling the policy in a direction that disagrees with the reward, a static μ over-commits to whichever side is currently less reliable.

We cast SFT–RL mixing as a minimum-mean-squared-error gradient estimation problem and derive a closed-form optimal mixing weight:

μ^* = ( α_tgt·Δg² + σ_r² ) / ( Δg² + σ_s² + σ_r² )

that balances SFT noise, RL noise, and SFT–RL disagreement. Because gradient-level quantities are prohibitively expensive at every step, we introduce three coefficient-space proxies — estimated online from tensors any GRPO/PPO trainer already computes — and wrap them in a guarded update stack: EMA smoothing, a cosine-schedule prior, per-step change capping, and hard clipping. The controller is length-invariant by construction and has less than 1% wall-time overhead.

Across math, code, science, and logic benchmarks, the noise-aware estimator alone outperforms the best rule-based baseline; the full system reaches +3.8 pp over HPT on average, while reducing KL-drift area by 28% and large |Δμ| events by more than 70%. Gains grow monotonically with model scale from Qwen2.5-1.5B to Qwen2.5-14B.

Key Highlights

Closed-form μ*

MSE-optimal derivation. No meta-learning, no bilevel optimization, no extra hyperparameter grid.

Coefficient-space proxies

Three cheap signals (σ_s², σ_r², Δg̃²) — any PPO / GRPO trainer already computes them.

Guarded controller

EMA smoothing → cosine-prior blend → per-step change cap → hard clipping. Robust to mid-training regime shifts.

< 1% wall-time overhead

Proxies computed inside the existing forward pass. No extra network eval, no shadow rollout.

Scales with model size

+2.2 pp @ 1.5B · +3.8 pp @ 7B · +3.3 pp @ 14B (AMC vs. HPT) — gains grow with model scale.

Length-invariant

Each sequence contributes exactly one scalar per proxy. Long rollouts don't get extra vote weight.

Results

Qwen2.5-7B-Instruct · mean ± std over 3 seeds · † = joint p<0.05 & Cohen's d>0.8 vs. best baseline

Table 1 · Math & knowledge reasoning

Method	AMC	AIME24	AIME25	MMLU-Pro
Qwen2.5-7B-Instruct	43.8	11.7	6.66	24.7
SFT-best	55.9 ±0.7	15.8 ±0.8	15.2 ±0.6	38.4 ±0.5
DPO	57.3 ±0.9	16.4 ±0.7	15.8 ±0.7	42.1 ±0.6
GRPO (pure RL)	52.1 ±1.4	13.2 ±1.1	8.54 ±1.0	45.8 ±0.9
CHORD	62.5 ±0.6	18.2 ±0.5	17.2 ±0.6	56.2 ±0.5
SRFT	61.8 ±0.7	17.9 ±0.6	17.0 ±0.7	55.6 ±0.5
LUFFY	63.1 ±0.6	18.5 ±0.5	17.6 ±0.6	56.0 ±0.5
HPT	63.4 ±0.5	18.7 ±0.5	17.8 ±0.6	56.4 ±0.4
KL-ctrl	62.8 ±0.8	18.4 ±0.6	17.6 ±0.7	55.8 ±0.6
GAC w/o φ	65.8 ±0.5	20.0 ±0.5	19.1 ±0.6	57.8 ±0.4
GAC + Token-φ (Ours)	67.2 ±0.4†	20.8 ±0.4†	19.8 ±0.5†	58.6 ±0.3†
Δ vs. best baseline (HPT)	+3.8	+2.1	+2.0	+2.2

Table 2 · Code generation (pass@1 %)

Method	MBPP	HumanEval	Avg.
Qwen2.5-7B-Instruct	68.4	72.0	70.2
CHORD	75.4 ±0.6	80.5 ±0.5	78.0
LUFFY	75.8 ±0.6	80.9 ±0.5	78.4
HPT	76.0 ±0.5	81.2 ±0.5	78.6
GAC + Token-φ	78.8 ±0.5†	83.5 ±0.4†	81.2
Δ vs. HPT	+2.8	+2.3	+2.6

Table 3 · Scaling — AMC accuracy across model sizes

Method	1.5B	7B	14B
CHORD	48.2 ±0.9	62.5 ±0.6	68.4 ±0.5
HPT	49.6 ±0.8	63.4 ±0.5	70.8 ±0.4
GAC w/o φ	51.4 ±0.8	65.8 ±0.5	73.2 ±0.4
GAC + Token-φ	51.8 ±0.7	67.2 ±0.4	74.1 ±0.4
Δ vs. HPT	+2.2	+3.8	+3.3

Gains grow with model size — larger models exhibit greater σ_s²/σ_r² dynamic range, giving the noise-aware controller more room to adapt.

Table 4 · Science & Logic — top-line vs. best baseline (HPT)

Domain	HPT	GAC + Token-φ	Δ
GPQA	40.4	43.5 ±0.5†	+3.1
SciBench	38.7	41.2 ±0.5†	+2.5
BBH-Logic (avg)	62.6	65.7 ±0.5†	+3.1

Evaluation performance and rollout dynamics — GAC consistently leads from ~200 steps and maintains a moderate response-length regime (~1.6-2.0k tokens), avoiding the 2.5-3.0k length spikes indicative of reward hacking in baselines.

Controller state — μ trajectory and the three coefficient-space proxies. μ starts near 0.85 (SFT-dominated), gradually decreases to ~0.15 as training matures, and tracks σ_r² rather than KL — the noise-aware estimator drives μ during >93% of steps.

How It Works

Each training step, the controller performs three operations:

① Proxy estimation. Three length-invariant scalars are computed from the batch:

σ_r² — variance of sequence-level GRPO-normalized advantages. Post-normalization dispersion, not raw reward variance.
σ_s² — tail-trimmed variance of length-normalized per-sequence NLL on expert samples.
Δg̃² — mean squared coefficient mismatch between SFT and RL token-level gradient coefficients on shared response tokens (after within-batch z-normalization).

② Closed-form aggregation into μ*:

μ^* = ( α_tgt · Δg̃² + σ_r² ) / ( Δg̃² + σ_s² + σ_r² )

③ Guarded update:

raw μ* → EMA(β) → blend(λ) with cosine prior → clip |Δμ| ≤ c̄ → clip to [μ_min, μ_max]

Detailed derivation (MSE minimization, proxy-vs-oracle validation, length-invariance proofs) is in Appendices A–D of the paper.

Code & Roadmap

github.com/deepnovacore/GAC

RELEASED v0.1.0 2026-08 · Reference AdaptiveMuController, hybrid-loss integration, unit tests, default config, and a public evaluator covering 11 task slices across 9 dataset families (Math / Knowledge / Code / Logic).
RELEASED v0.1.1 2026-09 · GAC-Qwen3.5-4B checkpoint release, fixed MMLU-Pro evaluation manifest, and dataset-aware benchmark documentation.
COMING v0.2.0 2026-09 · Training pipeline on top of VeRL, public wandb log links.
COMING v0.3.0 2026-10 · ModelScope mirror, gac-core on PyPI, and expanded external verification materials.
COMING v0.4.0 2026-11 · Docker image, 1-command reproduction, external verification runs.

BibTeX

@inproceedings{hu2026gac,
  title     = {GAC: Noise-Aware Adaptive Mixing for Hybrid SFT-RL Post-Training},
  author    = {Hu, Yuelin and Liu, Wei and Yu, Zhenbo and Cheng, Zhengxue and Song, Li},
  booktitle = {Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year      = {2026},
  address   = {Budapest, Hungary},
  url       = {https://openreview.net/forum?id=VhBpT4iq60},
}