Enhance machine reasoning via active learning with human rationales
Loading...
Authors
Yao, Bingsheng
Issue Date
2024-01
Type
Electronic thesis
Thesis
Thesis
Language
en_US
Keywords
Computer science
Alternative Title
Abstract
The main objective of this dissertation is to enhance model reasoning and explainability by introducing a new framework for Natural Language Generation (NLG). This framework generates rationales that are inspired by human reasoning, with the goal of providing insights into the underlying relationships present in the source data. By explicitly contributing to the final decision-making process, these rationales can help to improve the overall quality of the models. This process is implemented using an innovative Active Learning (AL) approach that incorporates human feedback on labels and rationales to select representative data to be annotated while minimizing the total amount of data annotation required. The proposed methodologies aim to address two limitations of current NLG advances, namely the lack of trustworthy model explainability and the scarcity of high-quality human feedback –- while transformer-based NLG models have demonstrated superior performance over various NLG tasks, they lack faithful explainability, which can lead to mistrust of their predictions. Additionally, high-quality human annotations are essential, but they are expensive and difficult to obtain in large quantities for many real-world domains. The dissertation proposes an approach to enhance the trustworthiness and explainability of NLG models. The approach is based on the human reasoning process, where intermediate rationales are generated about the source information first. These rationales are then served as guidance and explanations for the final predictions. The dissertation focuses on three crucial research questions to achieve this goal:
RQ 1. How can NLG models be guided to reason more comprehensible to humans?
RQ 2. Is the helpfulness of human rationales consistent with model predictions?
RQ 3. How to maximize the use of human feedback while minimizing annotation costs?
RQ 4. Can the proposed dual-model AL framework benefit from utilizing LLMs?
RQ 5. When do we need the proposed AL framework? As an initial step, this dissertation thoroughly evaluated the helpfulness of human rationales toward model predictions. The analysis was executed by evaluating the helpfulness in improving the prediction performance of NLG models at both the fine-tuning and inference stages, with a novel evaluation metric (RQ 1). The proposed framework was showcased through AL simulations on two NLG tasks with enhanced reasoning and explainability (RQ 2). The simulations provided evidence supporting the usefulness and efficiency, in terms of faster convergence and better performance, of incorporating human rationales in AL data selection with a novel AL sampling strategy (RQ 3). To extend the practical implication of the proposed framework with more advanced large language models (LLMs), this dissertation demonstrated the strengths and weaknesses of LLMs, such as FLAN-T5, in providing helpful rationales for predictions and resolving real-world NLG tasks. LLMs demonstrated strong potential for explanation generation in the proposed framework to further reduce human effort, despite LLMs suffering from low reliabilities for task-solving in real-world, domain-specific scenarios. The proposed framework could reduce annotation effort, maintain performance, and establish explainability in real-world tasks for NLG models (RQ 4). The final objective of the dissertation was to investigate the potential of the proposed AL framework to facilitate the development of domain-specific language models and large language models in real-world specialized domains.This dissertation executed a comprehensive benchmark and evaluation for state-of-the-art LLMs for six tasks in mental health detection via online posts. The evaluation results illustrated that traditional language models fine-tuned on specific tasks could perform comparable to generic LLMs, and multi-task fine-tuning for LLMs demonstrated promising results under low-resource scenarios. The observations shed light on the broad utility of the proposed AL framework to seamlessly support the development of domain-specific machine learning systems under low-resource scenarios, including task-specific fine-tuned traditional language models and instructional fine-tuned large language models (RQ 5). The proposed AL framework, which incorporates human feedback from labels and rationales, has the potential to facilitate a wide range of future research directions. The versatility of the proposed framework is manifested in improving the reliability of NLG models' task-solving in real-world scenarios and serving as the foundational learning framework for domain-specific NLG systems. Another promising future research direction is to develop LLM-simulated human agents and personalized assistants by actively fine-tuning LMs to simulate human behaviors and preferences. Moreover, a unified and automated annotation-learning framework could be achieved in the future, whereby the automated system automatically provides labels and rationales for unlabeled data, proactively queries humans for uncertain annotations, and iteratively learns from human feedback. This dissertation establishes the groundwork for more effective machine learning frameworks with faithful explainability, specialized NLG systems in real-world scenarios, and self-improving AI systems that can adjust to different preferences, such as personalized assistants.
Description
January2024
School of Science
School of Science
Full Citation
Publisher
Rensselaer Polytechnic Institute, Troy, NY