SSTAG: Structure-Aware Self-Supervised Learning Method for Text-Attributed Graphs

Yong Liu (Tsinghua University, Tsinghua University) · Xiaoshuai Hao (SAIT-China Lab, Samsung Research Center) · Ruyue Liu (Institute of Information Engineering Chinese Academy of Sciences) · Rong Yin (Institute of Information Engineering, Chinese Academy of Sciences) · Xiangzhen Bo (Wuhan University of Technology) · Jinwen Zhong (Institute of Information Engineering, Chinese Academy of Sciences) · Can Ma (Institute of Information Engineering, Chinese Academy of Sciences) · Weiping Wang (Institute of Information Engineering, CAS, China)

computer vision cross-domain generalization generalization ability graph learning graph neural networks graph representationsin-memory mechanismknowledge distillationmemory anchorsnatural language processing pre-trained models self-supervised learningstructure-aware multilayer perceptronstext-attributed graphs

Large-scale pre-trained models have revolutionized Natural Language Processing (NLP) and Computer Vision (CV), showcasing remarkable cross-domain generalization abilities. However, in graph learning, models are typically trained on individual graph datasets, limiting their capacity to transfer knowledge across different graphs and tasks. This approach also heavily relies on large volumes of annotated data, which presents a significant challenge in resource-constrained settings. Unlike NLP and CV, graph-structured data presents unique challenges due to its inherent heterogeneity, including domain-specific feature spaces and structural diversity across various applications. To address these challenges, we propose a novel structure-aware self-supervised learning method for Text-Attributed Graphs (SSTAG). By leveraging text as a unified representation medium for graph learning, SSTAG bridges the gap between the semantic reasoning of Large Language Models (LLMs) and the structural modeling capabilities of Graph Neural Networks (GNNs). Our approach introduces a dual knowledge distillation framework that co-distills both LLMs and GNNs into structure-aware multilayer perceptrons (MLPs), enhancing the scalability of large-scale TAGs. Additionally, we introduce an in-memory mechanism that stores typical graph representations, aligning them with memory anchors in an in-memory repository to integrate invariant knowledge, thereby improving the model’s generalization ability. Extensive experiments demonstrate that SSTAG outperforms state-of-the-art models on cross-domain transfer learning tasks, achieves exceptional scalability, and reduces inference costs while maintaining competitive performance.