Selective Learning for Deep Time Series Forecasting

Yisong Fu (Institute of Computing Technology, Chinese Academy of Sciences) · Fei Wang (Google) · Zezhi Shao (Institute of Computing Technology, Chinese Academy of Sciences) · Zhulin An (Institute of Computing Technology, Chinese Academy of Sciences) · Chengqing Yu (Institute of Computing Technology, Chinese Academy of Sciences) · Yujie Li (Chinese Academy of Sciences) · Yongjun Xu (Institute of Computing Technology, Chinese Academy of Sciences) · Qi Wang (Beijing Institute of Technology)
anomaly maskdual-mask mechanismgeneralizable timestepsinformeritransformermse lossoverfittingpredictive performanceresidual entropyresidual lower bound estimationselective learningstate-of-the-art modelstime series forecastingtimesnetuncertainty mask

Benefiting from high capacity for capturing complex temporal patterns, deep learning (DL) has significantly advanced time series forecasting (TSF). However, deep models tend to suffer from severe overfitting due to the inherent vulnerability of time series to noise and anomalies. The prevailing DL paradigm uniformly optimizes all timesteps through the MSE loss and learns those uncertain and anomalous timesteps without difference, ultimately resulting in overfitting. To address this, we propose a novel selective learning strategy for deep TSF. Specifically, selective learning screens a subset of the whole timesteps to calculate the MSE loss in optimization, guiding the model to focus on generalizable timesteps while disregarding non-generalizable ones. Our framework introduces a dual-mask mechanism to target timesteps: (1) an uncertainty mask leveraging residual entropy to filter uncertain timesteps, and (2) an anomaly mask employing residual lower bound estimation to exclude anomalous timesteps. Extensive experiments across eight real-world datasets demonstrate that selective learning can significantly improve the predictive performance for typical state-of-the-art deep models, including 37.4% MSE reduction for Informer, 8.4% for TimesNet, and 6.5% for iTransformer.