Yes, Transformers are Effective for Time Series Forecasting (+ Autoformer)

Abstraction: Autoformer transformer outperforms DLinear baseline for time series forecasting

Key points:

• Autoformer (NeurIPS 2021) adds two innovations to the transformer: a Decomposition Layer (separates trend and seasonality via moving average) and an Autocorrelation Mechanism replacing self-attention with FFT-based frequency-domain correlation at O(T log T) complexity
• DLinear (AAAI 2023) claimed to outperform transformers using only a linear layer on decomposed components; HuggingFace benchmarks show Autoformer (MASE 0.910) outperforms DLinear (MASE 0.965) on the Traffic dataset
• DLinear's weakness: no capacity to incorporate covariate/date-time features, causing it to miss weekend vs. weekday distributional shifts
• Univariate transformer models consistently outperform multivariate ones on smaller datasets due to spurious cross-series correlation estimation
• Autoformer is now available in HuggingFace Transformers as AutoformerModel
• Field bottleneck: lack of large-scale public time series datasets (analogous to ImageNet) limits pre-trained model development

Connections: Hugging Face · Transformers · Time Series Forecasting · Decomposition

Source: https://huggingface.co/blog/autoformer