Make LLMs' quotations more accurate and elegant !

QUILL: Quotation Generation Enhancement of Large Language Models

Jin Xiao¹, Bowei Zhang¹, Qianyu He¹, Jiaqing Liang*¹,
Feng Wei², Jinglei Chen², Zujie Liang², Deqing Yang¹, Yanghua Xiao¹

¹Fudan University ²MYbank, Ant Group

arXiv Code Dataset Demo Extension

🙋 While Large language models (LLMs) have become excellent writing assistants, they still struggle with Quotation Generation. This is because they either hallucinate when providing factual quotations or fail to provide quotes that exceed human expectations.

🤗 To bridge the gap, we first establish a holistic and automatic evaluation system for quotation generation task, which consists of five criteria each with corresponding automatic metric.

🔥 To improve the LLMs’ quotation generation abilities, we construct a bilingual knowledge base that is broad in scope and rich in dimensions, containing up to 32,022 quotes

🎉 Guided by our critiria, we further design a quotation-specific metric to rerank the retrieved quotations from the knowledge base. Extensive experiments show that our metrics strongly correlate with human preferences.

Framework

✍️ The framework for our Quotation Generation (QG) task research. We first establish an evaluation system with 5 evaluation criteria and automatic metrics, then build a quotation knowledge base covering multiple languages, topics and eras, and finally propose a quotation-specific reranking metric to rerank the quotations recalled in the RAG stage and improve the performance of QG tasks.

QUILL Copilo Chrome Extension

We have also developed quill extention, which is used for text polishing and rewriting with quotations. See the specific code. It is simple and easy to use. Everyone is welcome to use it!

BibTeX

@article{xiao24quill
  author    = {JinXiao, BoweiZhang, QianyuHe, JiaqingLiang, FengWei, JingleiChen, ZujieLiang, DeqingYang, YanghuaXiao},
  title     = {QUILL: Quotation Generation Enhancement of Large Language Models},
  year      = {2024},
}