Abstract
Large language models (LLMs), such as ChatGPT, are increasingly integrated into everyday applications across domains like education, business, and healthcare. Despite their growing prevalence, these models continue to exhibit several limitations, including inconsistency, factual inaccuracies, and hallucinations. To responsibly adopt such tools, it is essential to evaluate whether an LLM performs reliably for a specific use case. Evaluations should be used to guide which model to use or if a model should be used at all.
The foundation of any language model evaluation is the data that supports it. In this work, I introduce a three-step data-driven framework to better understand and improve evaluation approaches. (1) Data Acquisition - Selecting and collecting raw data. (2) Data Preparation - Preparing the raw data for evaluation through cleaning, labeling, transformation, and augmentation. (3) Data Validation - Ensuring the quality of the prepared data. Particularly, my work focuses on the data preparation stage of language model evaluation, which is often the most methodologically complex step and involves a fundamental tradeoff between resource costs and label quality. At one extreme, all data can be annotated manually, ensuring high-quality labels but at significant expense. At the other extreme, fully automated annotation methods can process data efficiently but risk producing inaccurate or low-quality labels.
My approach aims to strike a balance between these two ends of the spectrum by emphasizing automated methods that are strategically augmented with human expert knowledge. Through this hybrid approach, I improve data preparation by enhancing efficiency, enabling larger-scale annotation, and preserving user privacy. My research shows that when preparing data for language model evaluation, automated data processing methods should be utilized in conjunction with human expert knowledge to balance efficiency with quality.
