Sekai: A Video Dataset towards World Exploration

Zhen Li (Shenzhen Future Network of Intelligence Institute and Guangdong Provincial Key Laboratory of Future Networks of Intelligence, The Chinese University of Hong Kong (Shenzhen)) · Xiaofeng Mao (Alibaba Group) · Yuwei Wu (Beijing Institute of Technology) · Yunde Jia (Shenzhen MSU-BIT University) · Kaipeng Zhang (Shanghai AI Laboratory) · Chuanhao Li (Shanghai AI Laboratory) · Shaoheng Lin (South China University of Technology) · Ming Li (University of Central Florida) · Shitian Zhao · Zhaopan Xu (Harbin Institute of Technology) · Xinyue Li (Shanghai Artificial Intelligence Laboratory) · Yukang Feng (Nankai University) · Jianwen Sun (Nankai University) · Zizhen Li (Nankai University) · Fanrui Zhang (University of Science and Technology of China) · Jiaxin Ai (Wuhan University) · Zhixiang Wang (Tianjin University) · Tong He (Shanghai AI lab)
annotation qualitycamera trajectoriescomprehensive analysescrowd densitydataset limitationsdrone vieweffective toolboxfirst-person viewinteractive world explorationrich annotationsscene diversitytraining modelsvideo generationwalking viewweather conditionsworldwide video dataset

Video generation techniques have made remarkable progress, promising to be the foundation of interactive world exploration.However, existing video generation datasets are not well-suited for world exploration training as they suffer from some limitations: limited locations, short duration, static scenes, and a lack of annotations about exploration and the world.In this paper, we introduce Sekai (meaning "world" in Japanese), a high-quality first-person view worldwide video dataset with rich annotations for world exploration. It consists of over 5,000 hours of walking or drone view (FPV and UVA) videos from over 100 countries and regions across 750 cities. We develop an efficient and effective toolbox to collect, pre-process and annotate videos with location, scene, weather, crowd density, captions, and camera trajectories.Comprehensive analyses and experiments demonstrate the dataset’s scale, diversity, annotation quality, and effectiveness for training video generation models.We believe Sekai will benefit the area of video generation and world exploration, and motivate valuable applications.