Online Reinforcement Learning from Human Feedback with f-Divergence Regularization 37 views

Reinforcement Learning from Human Feedback (RLHF) has become a cornerstone technique for post-training large language models. While most existing approaches rely on the reverse KL-regularization, recent empirical studies have begun exploring alternative divergences (e.g., forward KL, chi-squared) as regularizers in RLHF. However, a unified theoretical understanding of general $f$-divergence regularization remains under-explored. To fill this gap, this thesis develops a theoretical framework for online RLHF under a general $f$-divergence regularized objective. We first characterize the optimal policy induced by a general divergence function and show how the choice of $f$ affects the sensitivity of the policy to reward perturbations. Building on this characterization, we study two exploration strategies for online preference learning. The first follows the principle of optimism in the face of uncertainty and constructs an exploration bonus adapted to pairwise preference feedback. The second introduces a derivative-based sampling method that uses the sensitivity of the $f$-regularized optimal policy as an implicit exploration signal. We provide unified theoretical guarantees for both algorithms. For the optimism-based method, we establish a logarithmic regret bound under general $f$-divergence regularization. For the derivative-based method, we prove an $O(1/T)$ suboptimality guarantee. Our analysis identifies regularizer-dependent constants that quantify how different choices of $f$ affect the performance bound. Numerical simulations support the theoretical findings and illustrate the behavior of the proposed algorithms across different divergence functions. Overall, this thesis provides a unified foundation for studying exploration and sample efficiency in $f$-divergence regularized RLHF.

MS (Master of Science)

English

All rights reserved by the author (no additional license for public reuse)

2026-07-16