Online Archive of University of Virginia Scholarship
Investigating Abstractive Summarization with Metric Reviews, Model Experiments, and a New Consistency Score119 views
Author
Ren, Yixuan, Computer Science - School of Engineering and Applied Science, University of Virginia
Advisors
Ji, Yangfeng, EN-Comp Science Dept, University of Virginia
Abstract
Despite their impressive performance in natural language generation tasks, Large Language Models (LLMs) still face critical challenges in text summarization. In particular, the performance of LLMs in abstractive text summarization and the limitations of existing evaluation frameworks warrant further investigation. In this work, we present a comprehensive analysis of summarization evaluation metrics, covering lexical overlap, semantic distance, factual consistency, and recent LLM-based methods. Employing these metrics as evaluation tools, we empirically assess the performance of summarization models across the LLaMA, and Gemma model families, utilizing datasets from diverse domains to provide an examination of the capabilities of current LLMs in abstractive text summarization tasks. To address limitations of current metrics, we introduce the concept of self-consistency and propose a novel consistency score to assess the reliability of text summarization models.
Degree
MS (Master of Science)
Keywords
Abstractive Text Summarization; Text Generation; Natural Language Processing
Ren, Yixuan. Investigating Abstractive Summarization with Metric Reviews, Model Experiments, and a New Consistency Score. University of Virginia, Computer Science - School of Engineering and Applied Science, MS (Master of Science), 2025-04-16, https://doi.org/10.18130/txax-4719.