FACT: Mitigating Inconsistent Hallucinations in LLMs via Fact-Driven Alternating Code-Text Training

Si Liu (Beihang University) · Xinxin You (Tsinghua University, Tsinghua University) · Qixin Sun (Beijing University of Aeronautics and Astronautics) · Chenwei Yan (University of International Business and Economics) · Xiao Zhang (University of Chicago) · Chen Ning (Tsinghua University) · Xiangling Fu (Beijing University of Posts and Telecommunications) · Guoping Hu (IFLYTEK CO.LTD.) · Shijin Wang (State Key Laboratory of Cognitive Intelligence) · Ji Wu (Tsinghua University) · Xien Liu ( Tsinghua University)

code-to-text predictionfact-based reasoningfact-driven alternating code-text traininginconsistent hallucinationslogical consistency numerical reasoning programming languagesqa and summarization tasksshared semantic spacesynthetic datasetstask-agnostic paradigmtask-specific trainingtext-to-code predictionwiki-40b-en

Inconsistent hallucinations remain a major challenge for large language models (LLMs), undermining the accuracy and reliability of fact-based reasoning in real-world applications. Existing approaches often rely on task-specific training or adaptation, such as hand-crafted synthetic datasets for domain tasks or solutions mainly focused on numerical reasoning, thereby limiting generalizability to broader, unseen NLP tasks. Inspired by the structural rigor and logical consistency of programming languages, we observe that fact-based texts can be mapped to programming structures due to their inherent patterns. We further propose FACT, a novel Fact-driven Alternating Code-text Training framework that alternates between text-to-code and code-to-text prediction. FACT is the first task-agnostic paradigm that embeds code and natural language in a shared semantic space, thereby transferring the logical consistency of code to LLM outputs in NLP tasks. Experiments show that with only a small subset of Wiki-40B-en for training, FACT reduces inconsistent hallucinations by 2.7%–8.0% and improves overall performance by 2.5%–6.1% in three leading LLMs and four diverse datasets covering QA and summarization tasks. This framework offers a new perspective on addressing challenging hallucinations in LLMs, contributing to more reliable AI.