Same Task, Different Circuits: Disentangling Modality-Specific Mechanisms in VLMs
accuracy gapcomputational sub-graphsexperimental tasksimage data representationsintervention strategymodality-specific data positionsmodel comparisonmulti-modal analysisperformance gapprocessing layerstask-specific circuitstextual representationstraining-free approachvision-language modelsvisual data tokens
Vision-Language models (VLMs) show impressive abilities to answer questions on visual inputs (e.g., counting objects in an image), yet demonstrate higher accuracies when performing an analogous task on text (e.g., counting words in a text). We investigate this accuracy gap by identifying and comparing the circuits