ChatbotID: Identifying Chatbots with Granger Causality Test

Yining Qi (Huazhong University of Science and Technology) · Haozhao Wang (The Hong Kong Polytechnic University) · Ruixuan Li (Huazhong University of Science and Technology) · Yuhua Li (Huazhong University of Science and Technology) · Xiaoquan Yi (Huazhong University of Science and Technology) · Wenchao Xu (Hong Kong University of Science and Technology) · Rui Zhang (Tsinghua University)

accuracy improvementsasymmetric patternbidirectional sentiment influencecontextual embeddingsconversational contextsdetection methodsdetection modelsgranger causality testhuman-chatbot interactionshuman-human interactionsinteractional featuresloss functionpredictive relationshipsreal-time dialogue

With the increasing sophistication of Large Language Models (LLMs), it is crucial to develop reliable methods to accurately identify whether an interlocutor in real-time dialogue is human or chatbot. However, existing detection methods are primarily designed for analyzing full documents, not the unique dynamics and characteristics of dialogue. These approaches frequently overlook the nuances of interaction that are essential in conversational contexts. This work identifies two key patterns in dialogues: (1) Human-Human (H-H) interactions exhibit significant bidirectional sentiment influence, while (2) Human-Chatbot (H-C) interactions display a clear asymmetric pattern. We propose an innovative approach named ChatbotID, which applies the Granger Causality Test (GCT) to extract a novel set of interactional features that capture the evolving, predictive relationships between conversational attributes. By synergistically fusing these GCT-based interactional features with contextual embeddings, and optimizing the model through a meticulous loss function. Experimental results across multiple datasets and detection models demonstrate the effectiveness of our framework, with significant improvements in accuracy for distinguishing between H-H and H-C dialogues.