Abstract
Large language models (LLMs) have shown impressive performance across a wide range of natural language tasks. In a traditional, model-centric view, architectural design and parameter scaling are often placed at the center of performance analysis and improvement. In a data-centric view, the focus shifts instead to understanding how the data at various stages of the model pipeline (e.g. training, adaptation, prompting, evaluation) impact the performance or understanding of a model.
In this dissertation, we develop data-centric methods for analyzing and improving LLMs across three stages of the model pipeline: fine-tuning, in-context learning, and pretraining-scale data analysis. First, we address the question of how to design scalable data contribution estimation (DCE) methods for LLM fine-tuning data. We motivate this section via foundational work on designing value functions for general machine learning models. Through a novel transferability task, we motivate and propose a scalable-by-design DCE method for LLMs. Second, as an alternative to fine-tuning, we address the question of how data impacts LLMs when learning without parameter updates in an in-context learning (ICL) setting. Specifically, we propose methods to analyze the performance impact of 1) the number of examples, and 2) input length as a statistical data bias. Finally, we address frontier cases for data attribution, focusing on scalable, trajectory-aware attribution for pretraining data and recursive synthetic-data feedback loops. The central claim of this dissertation is that data should be treated not only as an input to LLM development, but also as a measurable and actionable object of analysis.
