Aligning human and AI systems: framework, algorithm design and applications in large language models

Abstract

Aligning artificial intelligence (AI) systems with human values is essential to unlocking their full potential while ensuring ethical and equitable outcomes. As AI systems become increasingly integrated into our lives, it is crucial that they not only perform tasks effectively but also align with human preferences and values. This alignment helps prevent harm, mitigates biases, and ensures that AI systems serve humanity responsibly. My thesis develops a comprehensive framework of inverse reinforcement learning (IRL), which focuses on learning from the demonstration dataset. For my thesis work in inverse reinforcement learning, I will discuss problem formulation, algorithm design, theoretical analysis and also application of aligning large language models with demonstration dataset through IRL. The thesis is structured into four technical chapters (Chapter 2 to Chapter 5). The second chapter addresses the theoretical foundations of AI alignment, focusing on the inverse reinforcement learning (IRL) problem. IRL seeks to infer a reward function from expert demonstrations, which can then guide an agent's behavior. We propose novel IRL algorithms capable of learning from both expert demonstrations and online interactions with the environment, providing theoretical guarantees for convergence and optimality under specific conditions.The third chapter explores the challenges of IRL in scenarios where only offline datasets are available. We identify limitations in existing IRL approaches and introduce a new method tailored for offline learning. Theoretical guarantees for this approach are established, and its effectiveness is demonstrated through experiments on various tasks. The fourth chapter discusses the connection between the proposed algorithms and existing methods in the literature, which provides a foundational understanding of the theoretical underpinnings of our work. Moreover, the fourth chapter discuss how to combine the proposed algorithms with existing methods in the literature, highlighting the potential for leveraging complementary strengths to enhance AI alignment. The fifth chapter transitions to practical applications, focusing on aligning large language models (LLMs) with inverse reinforcement learning training pipelines. We adapt the proposed IRL algorithm to train LLMs, showcasing its effectiveness through extensive experiments. Additionally, we discuss the challenges, limitations, and future directions for research in AI alignment and reinforcement learning. The thesis concludes by summarizing the key contributions and findings throughout my PhD journey, emphasizing the importance of reinforcement learning in AI alignment and the need for continued research in this area.