Drift-Aware Multimodal User Representation Learning via Multi-Scale Temporal Modeling and Sparse Mixture-of-Experts
NeurIPS 2026 Β Β·Β [arXiv]
Understanding user preferences from noisy, temporally evolving social media behavior is challenging because of interest drift: preferences shift over time, exhibit both short- and long-term patterns, and span multiple co-existing interests.
DUMoE (Drift-aware User Modeling with Mixture-Of-Experts) is a unified framework for drift-aware multimodal user representation learning. It jointly models multimodal signals, multi-scale temporal dynamics, and multi-interest structure:
- β±οΈ Temporal dynamics-aware backbone β fuses static profile signals (MLP), short-term behavior (LSTM), and long-term dependencies (Transformer) into a coherent user vector
$z_u$ . - π§© Sparse MoE interest adapter β specialized experts model distinct interest subspaces; a gating network selects and aggregates a sparse
$\mathrm{Top}\text{-}K$ subset of experts to produce$h_u$ . - π― Three-stage training β Stage 1 backbone β Stage 2 expert specialization β Stage 3 gating, with multitask loss:
Figure 1. DUMoE overview: (a) temporal dynamics-aware backbone and (b) sparse MoE interest adapter, with dual heads for interest classification and interaction prediction.
| Module | What it does |
|---|---|
| Input | User profile |
| (a) Backbone | Static / short-term / long-term branches β fusion β |
| (b) MoE adapter | Gating + |
| Output | Classification head |
Paper main result = staged training (scripts/run_dumoe_staged.sh). Joint end-to-end training is provided as an ablation (scripts/run_dumoe_joint.sh).
Downstream evaluation freezes the encoder and probes:
| Task A β Interest classification | Task B β Interaction prediction | |
|---|---|---|
| Goal | Predict the 15-way interest of window |
Predict whether a candidate post is interacted in window |
| Input |
|
|
| Metrics | Hit@1, NDCG@3, Recall, KL | Acc, Recall, F1, BCE |
We release a large multimodal user dataset collected from X (formerly Twitter) via the official API (Nov 2025 β Mar 2026).
| Count | |
|---|---|
| Users | 14,015 |
| Posts | 7,685,700 |
| Posts with images | 2,890,668 |
| Interest domains | 15 |
Per-domain distribution of users, tweets, and images.
Public release = ids only (no post text, images, or usernames), following X redistribution rules. Unpack:
unzip dataset/ids/dumoe_ids.zip -d dataset/ids| File | Rows | Contents |
|---|---|---|
users.csv |
14,015 | numeric user_id |
posts.csv |
7,685,700 | user_id,post_id (newest β oldest) |
windows.csv |
156,757 | user_id,window_idx,label_id,split_tag |
Windows are 30 posts each (newest-first); split by index 0β7 train / 8β9 val / 10β11 test. Full docs, 15 labels, and preprocessing notes: dataset/README.md.
Questions about the dataset? Please open a GitHub issue.
Python >= 3.9
PyTorch >= 2.0
pip install -r requirements.txtgit clone https://github.com/idvxlab/DUMoE.git
cd DUMoE
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txtunzip dataset/ids/dumoe_ids.zip -d dataset/idsHydrate posts with the official X API if you need text/images, then build UNITE embeddings locally. Configure paths:
cp configs/paths.example.env configs/paths.env
# edit configs/paths.env β profile .pt, tweet emb shards, samples, labels cache
source configs/paths.envPlace large local files under dataset/local/ (gitignored).
bash scripts/run_dumoe_staged.shRuns Stage 1 β 2 β 3 with both heads (cls_weight=1.0, rec_weight=1.5). Hyperparameters: configs/paper_hparams.yaml.
Joint ablation (optional):
bash scripts/run_dumoe_joint.shbash scripts/run_baselines.shModels: SASRec, BERT4Rec, MIND, PeterRec, PTUM, UniSRec, HORAE, DisMIR, HM4SR.
bash scripts/run_downstream.sh # Task A then Task B
bash scripts/run_taskA.sh # 15-way window classification
bash scripts/run_taskB.sh # next-window interactionProtocol details: docs/downstream_protocol.md, downstream/README.md.
DUMoE/
βββ assets/ # figures (pipeline)
βββ dumoe/ # model + staged / joint training + ablations
βββ downstream/ # Task A (classify) / Task B (interaction)
βββ baselines/project/ # baseline pretraining + Exp1β3
βββ dataset/ # stats, labels, preprocessing, id zip
βββ scripts/ # one-command launchers
βββ configs/ # paper hparams + path template
βββ docs/ # Task A / Task B protocol
Not shipped: post text, images, usernames, UNITE embeddings, checkpoints, cookies, or API keys.
If you use this code or dataset, please cite:
@inproceedings{qian2026dumoe,
title={Drift-Aware Multimodal User Representation Learning via Multi-Scale Temporal Modeling and Sparse Mixture-of-Experts},
author={Qian, Ziqing and Chen, Haohang and Dang, Shengqi and Xiong, Yuhan and Shen, Canyu and Lei, Jiaying and Cao, Nan},
booktitle={Advances in Neural Information Processing Systems (NeurIPS)},
year={2026}
}Code is released under the MIT License.
Dataset ids are for research use; any hydrated X content remains under Xβs terms.