GAC: Noise-Aware Adaptive Mixing for Hybrid SFT-RL Post-Training

TL;DR

Hybrid SFT–RL post-training uses a fixed mixing weight μ that ignores how relative signal noise evolves during training. GAC casts μ as an MSE-optimal gradient estimation problem, gives a closed-form solution, and estimates it online from three cheap coefficient-space proxies. In the paper experiments: +3.8 pp over HPT on AMC (Qwen2.5-7B), consistent gains across math / code / science / logic, 28% less KL drift, and < 1% wall-time overhead.

GAC treats the SFT/RL mixing weight μ as an estimation target with a closed-form MSE-optimal solution. Three coefficient-space proxies (σ_s², σ_r², Δg̃²) feed a guarded update stack (EMA → cosine-prior blend → per-step cap → clipping).

Abstract

Hybrid post-training that combines supervised fine-tuning (SFT) and on-policy reinforcement learning (RL) is the standard recipe for aligning large language models. Yet the mixing weight μ that balances the two signals is almost always a fixed, hand-tuned schedule. This is fragile: when RL noise spikes, or when the SFT expert starts pulling the policy in a direction that disagrees with the reward, a static μ over-commits to whichever side is currently less reliable.

We cast SFT–RL mixing as a minimum-mean-squared-error gradient estimation problem and derive a closed-form optimal mixing weight:

μ^* = ( α_tgt·Δg² + σ_r² ) / ( Δg² + σ_s² + σ_r² )

that balances SFT noise, RL noise, and SFT–RL disagreement. Because gradient-level quantities are prohibitively expensive at every step, we introduce three coefficient-space proxies — estimated online from tensors any GRPO/PPO trainer already computes — and wrap them in a guarded update stack: EMA smoothing, a cosine-schedule prior, per-step change capping, and hard clipping. The controller is length-invariant by construction and has less than 1% wall-time overhead.

In the original paper experiments across math, code, science, and logic benchmarks, the noise-aware estimator alone outperforms the best rule-based baseline; the full system improves Qwen2.5-7B AMC accuracy by 3.8 percentage points over HPT, while reducing KL-drift area by 28% and large |Δμ| events by more than 70%. Gains are consistent across the evaluated Qwen2.5-1.5B, 7B, and 14B models.

Key Highlights

Closed-form μ*

MSE-optimal derivation. No meta-learning, no bilevel optimization, no extra hyperparameter grid.

Coefficient-space proxies

Three cheap signals (σ_s², σ_r², Δg̃²) — any PPO / GRPO trainer already computes them.

Guarded controller

EMA smoothing → cosine-prior blend → per-step change cap → hard clipping. Robust to mid-training regime shifts.

< 1% wall-time overhead

Proxies computed inside the existing forward pass. No extra network eval, no shadow rollout.

Scales with model size

+2.2 pp @ 1.5B · +3.8 pp @ 7B · +3.3 pp @ 14B (AMC vs. HPT) — gains grow with model scale.

Length-invariant

Each sequence contributes exactly one scalar per proxy. Long rollouts don't get extra vote weight.

GAC-Qwen3.5-4B Release

October 8, 2026 · Refreshed weights and full three-seed evaluation. We bring GAC to the Qwen3.5-4B backbone, combining hybrid SFT–RL post-training with reviewed mathematics and general-dialogue supervised refinement. Get the BF16 checkpoint on Hugging Face.

Mean over decoding seeds 0, 1, and 2 · evaluated October 3, 2026 · one checkpoint
Domain	Benchmark	Eval size per seed	GAC release mean
Mathematics	AMC	83	66.7%
Mathematics	AIME24	30	34.4%
Mathematics	AIME25	30	26.7%
Knowledge	MMLU-Pro	1,000	72.5%
Science	GPQA-Diamond	198	59.1%
Science	SciBench	692	58.5%
Code	MBPP	500	68.9%
Code	HumanEval	164	86.6%
Logic	BBH Logical Deduction	750	94.4%
Logic	BBH Object Counting	250	87.7%
Logic	BBH Tracking	750	95.6%
Logic	BBH average (macro)	3 slices / 1,750	92.6%

Protocol: 4,447 questions across 11 task slices; one answer per question per seed, temperature 0.6, top-p 0.95, and 8,192 new tokens for all slices in a joint batch per seed. Code scores are pass@1; BBH macro-average weights the three slices equally before rounding. These are decoding seeds, not independent training runs. Evaluation informed checkpoint development; this release is separate from the paper experiments.

Versioned evaluation profile and commands · Exact counts for all three seeds. General evaluator defaults differ from this release profile.

Original paper experiments · Qwen2.5 family

The following tables and figures report the original paper experiments, not the released Qwen3.5-4B checkpoint above.

Qwen2.5-7B-Instruct · mean ± std over 3 seeds · † = joint p<0.05 & Cohen's d>0.8 vs. best baseline

Table 1 · Math & knowledge reasoning

Method	AMC	AIME24	AIME25	MMLU-Pro
Qwen2.5-7B-Instruct	43.8	11.7	6.66	24.7
SFT-best	55.9 ±0.7	15.8 ±0.8	15.2 ±0.6	38.4 ±0.5
DPO	57.3 ±0.9	16.4 ±0.7	15.8 ±0.7	42.1 ±0.6
GRPO (pure RL)	52.1 ±1.4	13.2 ±1.1	8.54 ±1.0	45.8 ±0.9
CHORD	62.5 ±0.6	18.2 ±0.5	17.2 ±0.6	56.2 ±0.5
SRFT	61.8 ±0.7	17.9 ±0.6	17.0 ±0.7	55.6 ±0.5
LUFFY	63.1 ±0.6	18.5 ±0.5	17.6 ±0.6	56.0 ±0.5
HPT	63.4 ±0.5	18.7 ±0.5	17.8 ±0.6	56.4 ±0.4
KL-ctrl	62.8 ±0.8	18.4 ±0.6	17.6 ±0.7	55.8 ±0.6
GAC w/o φ	65.8 ±0.5	20.0 ±0.5	19.1 ±0.6	57.8 ±0.4
GAC + Token-φ (Ours)	67.2 ±0.4†	20.8 ±0.4†	19.8 ±0.5†	58.6 ±0.3†
Δ vs. best baseline (HPT)	+3.8	+2.1	+2.0	+2.2

Table 2 · Code generation (pass@1 %)

Method	MBPP	HumanEval	Avg.
Qwen2.5-7B-Instruct	68.4	72.0	70.2
CHORD	75.4 ±0.6	80.5 ±0.5	78.0
LUFFY	75.8 ±0.6	80.9 ±0.5	78.4
HPT	76.0 ±0.5	81.2 ±0.5	78.6
GAC + Token-φ	78.8 ±0.5†	83.5 ±0.4†	81.2
Δ vs. HPT	+2.8	+2.3	+2.6

Table 3 · Scaling — AMC accuracy across model sizes

Method	1.5B	7B	14B
CHORD	48.2 ±0.9	62.5 ±0.6	68.4 ±0.5
HPT	49.6 ±0.8	63.4 ±0.5	70.8 ±0.4
GAC w/o φ	51.4 ±0.8	65.8 ±0.5	73.2 ±0.4
GAC + Token-φ	51.8 ±0.7	67.2 ±0.4	74.1 ±0.4
Δ vs. HPT	+2.2	+3.8	+3.3

Original paper scaling experiment: GAC improves AMC accuracy over HPT by 2.2, 3.8, and 3.3 percentage points at 1.5B, 7B, and 14B respectively.

Table 4 · Science & Logic — top-line vs. best baseline (HPT)

Domain	HPT	GAC + Token-φ	Δ
GPQA	40.4	43.5 ±0.5†	+3.1
SciBench	38.7	41.2 ±0.5†	+2.5
BBH-Logic (avg)	62.6	65.7 ±0.5†	+3.1

Original paper evaluation metrics on the Qwen2.5 backbone

Evaluation performance and rollout dynamics — GAC consistently leads from ~200 steps and maintains a moderate response-length regime (~1.6-2.0k tokens), avoiding the 2.5-3.0k length spikes indicative of reward hacking in baselines.

Original paper GAC mixing-weight and proxy dynamics

Controller state — μ trajectory and the three coefficient-space proxies. μ starts near 0.85 (SFT-dominated), gradually decreases to ~0.15 as training matures, and tracks σ_r² rather than KL — the noise-aware estimator drives μ during >93% of steps.

How It Works

Each training step, the controller performs three operations:

① Proxy estimation. Three length-invariant scalars are computed from the batch:

σ_r² — variance of sequence-level GRPO-normalized advantages. Post-normalization dispersion, not raw reward variance.
σ_s² — tail-trimmed variance of length-normalized per-sequence NLL on expert samples.
Δg̃² — mean squared coefficient mismatch between SFT and RL token-level gradient coefficients on shared response tokens (after within-batch z-normalization).

② Closed-form aggregation into μ*:

μ^* = ( α_tgt · Δg̃² + σ_r² ) / ( Δg̃² + σ_s² + σ_r² )

③ Guarded update:

raw μ* → EMA(β) → blend(λ) with cosine prior → clip |Δμ| ≤ c̄ → clip to [μ_min, μ_max]

Detailed derivation (MSE minimization, proxy-vs-oracle validation, length-invariance proofs) is in Appendices A–D of the paper.

Code & Roadmap

github.com/deepnovacore/GAC

RELEASED v0.1.2 2026-10-08 · Refreshed GAC-Qwen3.5-4B weights, full three-decoding-seed results, and a frozen reproduction profile.
RELEASED v0.1.0 2026-08 · Reference AdaptiveMuController, hybrid-loss integration, unit tests, default config, and a public evaluator covering 11 task slices across 9 dataset families (Math / Knowledge / Code / Logic).
RELEASED v0.1.1 2026-09 · GAC-Qwen3.5-4B checkpoint release, fixed MMLU-Pro evaluation manifest, and dataset-aware benchmark documentation.
COMING v0.2.0 Planned · Public training-infrastructure packaging and reproducible recipe.
COMING v0.3.0 Planned · ModelScope mirror and expanded external verification materials.
COMING v0.4.0 Planned · Containerized reproduction and external verification runs.

BibTeX

@inproceedings{hu2026gac,
  title     = {GAC: Noise-Aware Adaptive Mixing for Hybrid SFT-RL Post-Training},
  author    = {Hu, Yuelin and Liu, Wei and Yu, Zhenbo and Cheng, Zhengxue and Song, Li},
  booktitle = {Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year      = {2026},
  address   = {Budapest, Hungary},
  url       = {https://openreview.net/forum?id=VhBpT4iq60},
}