EgoBlind: Towards Egocentric Visual Assistance for the Blind

Zhulin Tao (Communication University of China) · Xun Yang (University of Science and Technology of China) · Meng Wang (Hefei University of Technology) · Hao Qiu (University of Milan) · Richang Hong (Hefei University of Technology) · Junbin Xiao (National University of Singapore) · Nanxin Huang (Communication University of China) · Angela Yao (National University of Singapore)
accuracy evaluationai assistantsassistive capabilitiesdataset collectionegoblindegocentric videoqafirst-person videosheuristic solutionshuman performanceindependence enhancementlimitationsmanual annotationmultimodal large language modelsreference answersvisual assistancevisually impaired

We present EgoBlind, the first egocentric VideoQA dataset collected from blind individuals to evaluate the assistive capabilities of contemporary multimodal large language models (MLLMs). EgoBlind comprises 1,392 first-person videos from the daily lives of blind and visually impaired individuals. It also features 5,311 questions directly posed or verified by the blind to reflect their in-situation needs for visual assistance. Each question has an average of 3 manually annotated reference answers to reduce subjectiveness.Using EgoBlind, we comprehensively evaluate 16 advanced MLLMs and find that all models struggle. The best performers achieve an accuracy near 60\%, which is far behind human performance of 87.4\%. To guide future advancements, we identify and summarize major limitations of existing MLLMs in egocentric visual assistance for the blind and explore heuristic solutions for improvement. With these efforts, we hope that EgoBlind will serve as a foundation for developing effective AI assistants to enhance the independence of the blind and visually impaired. Data and code are available at \url{https://github.com/doc-doc/EgoBlind}.