On the current landscape of language model reward modeling for alignment

Abstract

Reinforcement Learning with Human Feedback (RLHF) has become a central approach for aligning large language models (LLMs) with human preferences. Despite its success, a significant gap remains between human judgment and LLM behavior, particularly in automated evaluation settings and in capturing the multi-dimensional nature of human preferences. Furthermore, pinpointing where alignment methods fail to accurately reflect the true distribution of human preferences remains a challenge, largely due to the sparse and coarse nature of scalar reward signals typically employed in RLHF. This thesis investigates several dimensions of the alignment problem to bridge the behavioral gap between human judgments and LLM outputs. Specifically, we examine (i) the limitations of AI-based alignment approaches that use LLMs as evaluators,(ii) dynamic alignment strategies that incorporate multiple aspects of human judgments, and (iii) dense alignment techniques that reshape token-level reward distributions via Bayesian optimization. Through rigorous stress testing of LLM-based evaluations and the development of novel alignment methods, this work advances the field by enabling more nuanced and robust modeling of human preferences.