STaRFormer: Semi-Supervised Task-Informed Representation Learning via Dynamic Attention-Based Regional Masking for Sequential Data

Maximilian Forstenhäusler (University of Glasgow) · Daniel Külzer (BMW Group) · Christos Anagnostopoulos (University of Glasgow) · Shameem Puthiya Parambath (University of Glasgow) · Natascha Weber (BMW Deutschland)

comprehensive experimentscontext-aware decision-makingdynamic attentionirregularly sampled datalatent representations non-stationary dataregional maskingsemi-supervised contrastive learningsequential modelingsequential spatiotemporal datasituational awarenessstate-of-the-art approachestask-specific enhancementstransformer-based approachuser intent

Understanding user intent is essential for situational and context-aware decision-making. Motivated by a real-world scenario, this work addresses intent predictions of smart device users in the vicinity of vehicles by modeling sequential spatiotemporal data. However, in real-world scenarios, environmental factors and sensor limitations can result in non-stationary and irregularly sampled data, posing significant challenges. To address these issues, we propose STaRFormer, a Transformer-based approach that can serve as a universal framework for sequential modeling. STaRFormer utilizes a new dynamic attention-based regional masking scheme combined with a novel semi-supervised contrastive learning paradigm to enhance task-specific latent representations. Comprehensive experiments on 56 datasets varying in types (including non-stationary and irregularly sampled), tasks, domains, sequence lengths, training samples, and applications demonstrate the efficacy of STaRFormer, achieving notable improvements over state-of-the-art approaches.