Security Challenges in AI Agent Deployment: Insights from a Large Scale Public Competition

Andy Zou (CMU, Gray Swan AI) · Maxwell Lin (University of California, Berkeley) · Eliot Jones (Gray Swan) · Micha Nowak (Bayerische Julius-Maximilians-Universität Würzburg) · Mateusz Dziemian (Independent) · Nick Winter (Gray Swan AI) · Valent Nathanael (Gray Swan AI) · Ayla Croft (Gray Swan AI) · Xander Davies (University of Oxford) · Jai Patel (UK AI Security Institute) · Robert Kirk (University College London) · Yarin Gal (University of Oxford) · Dan Hendrycks (Center for AI Safety) · Zico Kolter (Carnegie Mellon University) · Matt Fredrikson (CMU)
agent robustnessai agent vulnerabilitiesattack success ratesbenchmark constructioncritical vulnerabilitiesdefense improvementsdeployment policiesfrontier llmsinference compute budgetmodel capabilitypolicy violationsprompt injection attacksred teamingtransferabilityuniversality

AI agents are rapidly being deployed across diverse industries, but can they adhere to deployment policies under attacks? We organized a one-month red teaming challenge