V-CECE: Visual Counterfactual Explanations via Conceptual Edits

Nikolaos Spanos (National Technical University of Athens University of Pennsylvania) · Maria Lymperaiou (National Technical University of Athens) · Giorgos Filandrianos (National Technical University of Athens) · Konstantinos Thomas (National Technical University of Athens) · Athanasios Voulodimos (National Technical University of Athens) · Giorgos Stamou (National Technical University of Athens)

black-box counterfactual generationcomprehensive human evaluationconvolutional neural network (cnn)explainable counterfactual generationexplanatory gaphuman reasoninghuman-level counterfactual explanationslarge vision language model (lvlm)neural model behavioroptimal editspre-trained image editing diffusion modelsemantic content theoretical guaranteesvision transformer (vit)zero training

Recent black-box counterfactual generation frameworks fail to take into account the semantic content of the proposed edits, while relying heavily on training to guide the generation process. We propose a novel, plug-and-play black-box counterfactual generation framework, which suggests step-by-step edits based on theoretical guarantees of optimal edits to produce human-level counterfactual explanations with zero training. Our framework utilizes a pre-trained image editing diffusion model, and operates without access to the internals of the classifier, leading to an explainable counterfactual generation process. Throughout our experimentation, we showcase the explanatory gap between human reasoning and neural model behavior by utilizing both Convolutional Neural Network (CNN), Vision Transformer (ViT) and Large Vision Language Model (LVLM) classifiers, substantiated through a comprehensive human evaluation.