Evaluating LLM-contaminated Crowdsourcing Data Without Ground Truth
annotation taskscrowdsourcingempirical demonstrationgenerative aihuman feedbackllm collusionllm detectionlow-effort cheatingpeer predictionreal-world datasetsstructured annotationtheoretical guaranteestraining-free scoringtrustworthy ai
The recent success of generative AI highlights the crucial role of high-quality human feedback in building trustworthy AI systems. However, the increasing use of large language models (LLMs) by crowdsourcing workers poses a significant challenge: datasets intended to reflect human input may be compromised by LLM-generated responses. Existing LLM detection approaches often rely on high-dimensional training data such as text, making them unsuitable for structured annotation tasks like multiple-choice labeling. In this work, we investigate the potential of peer prediction