Addressing investor concerns: a Chinese financial question-answering benchmark with LLM-based evaluation

In recent years, large language models (LLMs) have shown impressive performance across various natural language processing tasks and are increasingly adopted in high-stakes fields such as financial analysis. However, their effectiveness in Chinese financial contexts is hindered by the scarcity of high-quality, domain-specific datasets. To bridge this gap, we present the Chinese Financial Question Answering (CFQA) dataset, a novel resource designed to advance research in financial analysis. CFQA is constructed from publicly available annual reports of multiple Chinese listed companies, paired with corresponding questions and human-annotated answers. Evaluation results reveal that existing QA methods perform poorly on this dataset. CFQA introduces several unique challenges: (1) source documents are in PDF format with complex tabular structures, making information extraction difficult; (2) the length and intricacy of financial reports complicate answer retrieval; and (3) the questions are tightly focused on domain-specific financial content.