New Go-playing trick defeats world-class Go AI—but loses to human amateurs

Abstraction: Adversarial policy exploits off-distribution moves to beat world-class Go AI

Key points:

• UC Berkeley researchers developed an adversarial policy (neural network + Monte Carlo Tree Search) that consistently defeats KataGo despite being beatable by amateur human players
• The strategy claims a small corner of the board, concedes most territory to KataGo, then plays seemingly easy-to-capture stones — exploiting KataGo's inability to evaluate this unusual off-distribution pattern
• KataGo's vulnerability stems from training distribution limits: "generalizes well to many novel strategies, but gets weaker the further away it gets from the games it saw during training"
• The adversarial policy works without any search at test time against KataGo, showing deep-learning Go AI has exploitable blind spots despite superhuman average performance
• Finding illustrates a general challenge: self-play training can produce very strong but brittle agents with exploitable gaps in coverage
• Research conducted by Adam Gleave, Tony Wang, Nora Belrose et al., affiliated with UC Berkeley and the Alignment Forum

Connections: Katago · Alphago · Deepmind · Adversarial Examples · Reinforcement Learning · AI Safety

Source: https://arstechnica.com/information-technology/2022/11/new-go-playing-trick-defeats-world-class-go-ai-but-loses-to-human-amateurs/