skip to main content
research-article

PerFedRec++: Enhancing Personalized Federated Recommendation with Self-Supervised Pre-Training

Published: 05 November 2024 Publication History

Abstract

Federated recommendation systems employ federated learning techniques to safeguard user privacy by transmitting model parameters instead of raw user data between user devices and the central server. Nevertheless, the current federated recommender system faces three significant challenges: (1) data heterogeneity: the heterogeneity of users’ attributes and local data necessitates the acquisition of personalized models to improve the performance of federated recommendation; (2) model performance degradation: the privacy-preserving protocol design in the federated recommendation, such as pseudo item labeling and differential privacy, would deteriorate the model performance; (3) communication bottleneck: the standard federated recommendation algorithm can have a high communication overhead. Previous studies have attempted to address these issues, but none have been able to solve them simultaneously. In this article, we propose a novel framework, named PerFedRec++, to enhance the personalized federated recommendation with self-supervised pre-training. Specifically, we utilize the privacy-preserving mechanism of federated recommender systems to generate two augmented graph views, which are used as contrastive tasks in self-supervised graph learning to pre-train the model. Pre-training enhances the performance of federated models by improving the uniformity of representation learning. Also, by providing a better initial state for federated training, pre-training makes the overall training converge faster, thus alleviating the heavy communication burden. We then construct a collaborative graph to learn the client representation through a federated graph neural network. Based on these learned representations, we cluster users into different user groups and learn personalized models for each cluster. Each user learns a personalized model by combining the global federated model, the cluster-level federated model, and its own fine-tuned local model. Experiments on three real-world datasets show that our proposed method achieves superior performance over existing methods.

References

[1]
Waqar Ali, Rajesh Kumar, Zhiyi Deng, Yansong Wang, and Jie Shao. 2021. A federated learning approach for privacy protection in context-aware recommender systems. The Computer Journal 64, 7 (2021), 1016–1027.
[2]
Muhammad Ammad-Ud-Din, Elena Ivannikova, Suleiman A. Khan, Were Oyomno, Qiang Fu, Kuan Eeik Tan, and Adrian Flanagan. 2019. Federated collaborative filtering for privacy-preserving personalized recommendation system. arXiv:1901.09888. Retrieved from
[3]
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. 2021. Beit: Bert pre-training of image transformers. arXiv:2106.08254. Retrieved from
[4]
Rianne van den Berg, Thomas N. Kipf, and Max Welling. 2017. Graph convolutional matrix completion. arXiv:1706.02263. Retrieved from ArXiv abs/1706.02263
[5]
Di Chai, Leye Wang, Kai Chen, and Qiang Yang. 2020. Secure federated matrix factorization. IEEE Intelligent Systems 36, 5 (2020), 11–20.
[6]
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020b. A simple framework for contrastive learning of visual representations. In Proceedings of the International Conference on Machine Learning. PMLR, 1597–1607.
[7]
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. 2020a. Improved baselines with momentum contrastive learning. arXiv:2003.04297. Retrieved from
[8]
Alexis Conneau and Guillaume Lample. 2019. Cross-lingual language model pretraining. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 32, 7057–7067.
[9]
Mukund Deshpande and George Karypis. 2004. Item-based top-n recommendation algorithms. ACM Transactions on Information Systems (TOIS) 22, 1 (2004), 143–177.
[10]
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805. Retrieved from
[11]
Yongjie Du, Deyun Zhou, Yu Xie, Jiao Shi, and Maoguo Gong. 2021. Federated matrix factorization for privacy-preserving recommender systems. Applied Soft Computing 111 (2021), 107700.
[12]
Cynthia Dwork. 2006. Differential privacy. In Proceedings of the Automata, Languages and Programming: 33rd International Colloquium (ICALP ’06) (Mar 1997), 57–98. DOI:
[13]
Cynthia Dwork. 2008. Differential privacy: A survey of results. In Proceedings of the International Conference on Theory and Applications of Models of Computation. Springer, 1–19.
[14]
Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. 2020. Personalized federated learning: A meta-learning approach. arXiv:2002.07948. Retrieved from
[15]
John Giorgi, Osvald Nitski, Bo Wang, and Gary Bader. 2020. DeCLUTR: Deep contrastive learning for unsupervised textual representations. arXiv:2006.03659. Retrieved from
[16]
Yu Guo, Hongcheng Xie, Yinbin Miao, Cong Wang, and Xiaohua Jia. 2020. FedCrowd: A federated and privacy-preserving crowdsourcing platform on blockchain. IEEE Transactions on Services Computing 15, 4 (2020), 2060–2073.
[17]
Michael Gutmann and Aapo Hyvärinen. 2010. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the 13th International Conference on Artificial Intelligence and Statistics. JMLR Workshop and Conference Proceedings, 297–304.
[18]
Ruining He and Julian McAuley. 2016. Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering. In Proceedings of the 25th International Conference on World Wide Web. 507–517.
[19]
Xiangnan He, Kuan Deng, Xiang Wang, Yan Li, Yongdong Zhang, and Meng Wang. 2020. LightGCN: Simplifying and powering graph convolution network for recommendation. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. 639–648.
[20]
Ashish Jaiswal, Ashwin Ramesh Babu, Mohammad Zaki Zadeh, Debapriya Banerjee, and Fillia Makedon. 2020. A survey on contrastive self-supervised learning. Technologies 9, 1 (2020), 2.
[21]
Longlong Jing and Yingli Tian. 2020. Self-supervised visual feature learning with deep neural networks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence 43, 11 (2020), 4037–4058.
[22]
Yehuda Koren, Robert Bell, and Chris Volinsky. 2009. Matrix factorization techniques for recommender systems. Computer 42, 8 (2009), 30–37.
[23]
Zitao Li, Bolin Ding, Ce Zhang, Ninghui Li, and Jingren Zhou. 2021. Federated matrix factorization with privacy guarantee. Proceedings of the VLDB Endowment 15, 4 (2021), 900–913.
[24]
Dawen Liang, Laurent Charlin, James McInerney, and David M Blei. 2016. Modeling user exposure in recommendation. In Proceedings of the 25th International Conference on World Wide Web. 951–961.
[25]
Yujie Lin, Pengjie Ren, Zhumin Chen, Zhaochun Ren, Dongxiao Yu, Jun Ma, Maarten de Rijke, and Xiuzhen Cheng. 2020. Meta matrix factorization for federated rating predictions. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. 981–990.
[26]
Xiao Liu, Fanjin Zhang, Zhenyu Hou, Li Mian, Zhaoyu Wang, Jing Zhang, and Jie Tang. 2021b. Self-supervised learning: Generative or contrastive. IEEE Transactions on Knowledge and Data Engineering 35, 1 (2021), 857–876.
[27]
Zhiwei Liu, Liangwei Yang, Ziwei Fan, Hao Peng, and Philip S. Yu. 2021a. Federated social recommendation with graph neural network. arXiv:2111.10778. Retrieved from
[28]
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. ViLBERT: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 32, 13–23.
[29]
Sichun Luo, Bowei He, Haohan Zhao, Yinya Huang, Aojun Zhou, Zongpeng Li, Yuanzhang Xiao, Mingjie Zhan, and Linqi Song. 2023a. RecRanker: Instruction tuning large language model as ranker for top-k recommendation. 8024–8035. arXiv:2312.16018. Retrieved from
[30]
Sichun Luo, Chen Ma, Yuanzhang Xiao, and Linqi Song. 2023b. Improving long-tail item recommendation with graph augmentation. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management. 1707–1716.
[31]
Sichun Luo, Jiansheng Wang, Aojun Zhou, Li Ma, and Linqi Song. 2024a. Large language models augmented rating prediction in recommender system. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP ’24). IEEE, 7960–7964.
[32]
Sichun Luo, Yuanzhang Xiao, Yang Liu, Congduan Li, and Linqi Song. 2022b. Towards communication efficient and fair federated personalized sequential recommendation. In Proceedings of the 5th International Conference on Information Communication and Signal Processing (ICICSP ’22). IEEE, 1–6.
[33]
Sichun Luo, Yuanzhang Xiao, and Linqi Song. 2022a. Personalized federated recommendation via joint representation learning, user clustering, and model adaptation. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management. 4289–4293.
[34]
Sichun Luo, Yuxuan Yao, Bowei He, Yinya Huang, Aojun Zhou, Xinyi Zhang, Yuanzhang Xiao, Mingjie Zhan, and Linqi Song. 2024b. Integrating large language models into recommendation via mutual augmentation and adaptive aggregation. arXiv:2401.13870. Retrieved from
[35]
Sichun Luo, Xinyi Zhang, Yuanzhang Xiao, and Linqi Song. 2022c. HySAGE: A hybrid static and adaptive graph embedding network for context-drifting recommendations. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management. 1389–1398.
[36]
James MacQueen. 1967. Some methods for classification and analysis of multivariate observations. In Proceedings of the 5th Berkeley Symposium on Mathematical Statistics and Probability, Vol. 1. 281–297.
[37]
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. 2017. Communication-efficient learning of deep networks from decentralized data. In Proceedings of the Artificial Intelligence and Statistics. PMLR, 1273–1282.
[38]
Lorenzo Minto, Moritz Haller, Benjamin Livshits, and Hamed Haddadi. 2021. Stronger privacy for federated collaborative filtering with implicit feedback. In Proceedings of the 15th ACM Conference on Recommender Systems. 342–350.
[39]
Ishan Misra and Laurens van der Maaten. 2020. Self-supervised learning of pretext-invariant representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 6707–6717.
[40]
Khalil Muhammad, Qinqin Wang, Diarmuid O’Reilly-Morgan, Elias Tragos, Barry Smyth, Neil Hurley, James Geraci, and Aonghus Lawlor. 2020. FedFast: Going beyond average for faster training of federated recommender systems. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 1234–1242.
[41]
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. PyTorch: An imperative style, high-performance deep learning library. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 32, 8024–8035.
[42]
Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. 2012. BPR: Bayesian personalized ranking from implicit feedback. arXiv:1205.2618. Retrieved from
[43]
Stefano Savazzi, Monica Nicoli, Mehdi Bennis, Sanaz Kianoush, and Luca Barbieri. 2021. Opportunities of federated learning in connected, cooperative, and automated industrial systems. IEEE Communications Magazine 59, 2 (2021), 16–21.
[44]
Zehua Sun, Yonghui Xu, Yong Liu, Wei He, Yali Jiang, Fangzhao Wu, and Lizhen Cui. 2022. A survey on federated recommendation systems. arXiv:2301.00767. Retrieved from
[45]
Ben Tan, Bo Liu, Vincent Zheng, and Qiang Yang. 2020. A federated recommender system for online services. In Proceedings of the 14th ACM Conference on Recommender Systems. 579–581.
[46]
Qinyong Wang, Hongzhi Yin, Tong Chen, Zi Huang, Hao Wang, Yanchang Zhao, and Nguyen Quoc Viet Hung. 2020. Next point-of-interest recommendation on resource-constrained mobile devices. In Proceedings of the Web Conference 2020. 906–916.
[47]
Qinyong Wang, Hongzhi Yin, Tong Chen, Junliang Yu, Alexander Zhou, and Xiangliang Zhang. 2022. Fast-adapting and privacy-preserving federated recommender system. The VLDB Journal 31, 5 (2022), 877–896.
[48]
Tongzhou Wang and Phillip Isola. 2020. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In Proceedings of the International Conference on Machine Learning. PMLR, 9929–9939.
[49]
Xiang Wang, Xiangnan He, Meng Wang, Fuli Feng, and Tat-Seng Chua. 2019. Neural graph collaborative filtering. In Proceedings of the 42nd international ACM SIGIR conference on Research and development in Information Retrieval. 165–174.
[50]
Chuhan Wu, Fangzhao Wu, Yang Cao, Yongfeng Huang, and Xing Xie. 2021c. FedGNN: Federated graph neural network for privacy-preserving recommendation. arXiv:2102.04925. Retrieved from
[51]
Chuhan Wu, Fangzhao Wu, Tao Qi, Yongfeng Huang, and Xing Xie. 2022b. FedCL: Federated contrastive learning for privacy-preserving recommendation. arXiv:2204.09850. Retrieved from
[52]
Jinze Wu, Qi Liu, Zhenya Huang, Yuting Ning, Hao Wang, Enhong Chen, Jinfeng Yi, and Bowen Zhou. 2021a. Hierarchical personalized federated learning for user modeling. In Proceedings of the Web Conference 2021. 957–968.
[53]
Jiancan Wu, Xiang Wang, Fuli Feng, Xiangnan He, Liang Chen, Jianxun Lian, and Xing Xie. 2021b. Self-supervised graph learning for recommendation. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. 726–735.
[54]
Shiwen Wu, Fei Sun, Wentao Zhang, Xu Xie, and Bin Cui. 2022a. Graph neural networks in recommender systems: A survey. ACM Computing Surveys 55, 5 (2022), 1–37.
[55]
Xu Xie, Fei Sun, Zhaoyang Liu, Shiwen Wu, Jinyang Gao, Jiandong Zhang, Bolin Ding, and Bin Cui. 2022a. Contrastive learning for sequential recommendation. In Proceedings of the IEEE 38th International Conference on Data Engineering (ICDE ’22). IEEE, 1259–1273.
[56]
Yaochen Xie, Zhao Xu, Jingtun Zhang, Zhengyang Wang, and Shuiwang Ji. 2022b. Self-supervised learning of graph neural networks: A unified review. IEEE Transactions on Pattern Analysis and Machine Intelligence 45, 2 (2022), 2412–2429.
[57]
Jian Xu, Shao-Lun Huang, Linqi Song, and Tian Lan. 2022a. Byzantine-robust federated learning through collaborative malicious gradient filtering. In Proceedings of the IEEE 42nd International Conference on Distributed Computing Systems (ICDCS ’22). IEEE, 1223–1235.
[58]
Jian Xu, Xinyi Tong, and Shao-Lun Huang. 2022b. Personalized federated learning with feature alignment and classifier collaboration. In Proceedings of the 11th International Conference on Learning Representations. 1–25.
[59]
Jian Xu, Yi Yan, and Shao-Lun Huang. 2022c. FedPer++: Toward improved personalized federated learning on heterogeneous and imbalanced data. In Proceedings of the International Joint Conference on Neural Networks (IJCNN ’22). IEEE, 01–08.
[60]
Liu Yang, Ben Tan, Vincent W. Zheng, Kai Chen, and Qiang Yang. 2020. Federated recommendation systems. In Federated Learning: Privacy and Incentive, Springer, 225–239.
[61]
Qiang Yang, Yang Liu, Tianjian Chen, and Yongxin Tong. 2019. Federated machine learning: Concept and applications. ACM Transactions on Intelligent Systems and Technology (TIST) 10, 2 (2019), 1–19.
[62]
Yelp. 2018. Yelp Dataset Challenge. Retrieved from https://www.yelp.com/dataset/challenge
[63]
Rex Ying, Ruining He, Kaifeng Chen, Pong Eksombatchai, William L Hamilton, and Jure Leskovec. 2018. Graph convolutional neural networks for web-scale recommender systems. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 974–983.
[64]
Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. 2020. Graph contrastive learning with augmentations. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 33. 5812–5823.
[65]
Junliang Yu, Xin Xia, Tong Chen, Lizhen Cui, Nguyen Quoc Viet Hung, and Hongzhi Yin. 2022a. XSimGCL: Towards extremely simple graph contrastive learning for recommendation. arXiv:2209.02544.
[66]
Junliang Yu, Hongzhi Yin, Xin Xia, Tong Chen, Lizhen Cui, and Quoc Viet Hung Nguyen. 2022b. Are graph augmentations necessary? Simple graph contrastive learning for recommendation. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval. 1294–1303.
[67]
Chen Zhang, Yu Guo, Hongwei Du, and Xiaohua Jia. 2020. PFcrowd: Privacy-preserving and federated crowdsourcing framework by using blockchain. In Proceedings of the 2020 IEEE/ACM 28th International Symposium on Quality of Service (IWQoS). IEEE, 1–10.
[68]
Honglei Zhang, Fangyuan Luo, Jun Wu, Xiangnan He, and Yidong Li. 2023. LightFR: Lightweight federated recommendation with privacy-preserving matrix factorization. ACM Transactions on Information Systems 41, 4 (2023), 1–28.
[69]
Huasong Zhong, Jianlong Wu, Chong Chen, Jianqiang Huang, Minghua Deng, Liqiang Nie, Zhouchen Lin, and Xian-Sheng Hua. 2021. Graph contrastive clustering. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 9224–9233.
[70]
Xin Zhou, Aixin Sun, Yong Liu, Jie Zhang, and Chunyan Miao. 2023. SelfCF: A simple framework for self-supervised collaborative filtering. ACM Transactions on Recommender Systems 1, 2 (2023), 1–25.

Cited By

View all
  • (2025)Privacy-preserved and Responsible Recommenders: From Conventional Defense to Federated Learning and BlockchainACM Computing Surveys10.1145/370898257:5(1-35)Online publication date: 9-Jan-2025
  • (2024)Federated Learning-Based Social Recommendation with Social Denoising2024 International Conference on New Trends in Computational Intelligence (NTCI)10.1109/NTCI64025.2024.10776360(331-335)Online publication date: 18-Oct-2024
  • (2024)Large Language Models Augmented Rating Prediction in Recommender SystemICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)10.1109/ICASSP48485.2024.10447514(7960-7964)Online publication date: 14-Apr-2024

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Intelligent Systems and Technology
ACM Transactions on Intelligent Systems and Technology  Volume 15, Issue 5
October 2024
719 pages
EISSN:2157-6912
DOI:10.1145/3613688
  • Editor:
  • Huan Liu
Issue’s Table of Contents

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 05 November 2024
Online AM: 14 May 2024
Accepted: 22 April 2024
Revised: 04 February 2024
Received: 11 May 2023
Published in TIST Volume 15, Issue 5

Check for updates

Author Tags

  1. Federated learning
  2. self-supervised learning
  3. personalization

Qualifiers

  • Research-article

Funding Sources

  • National Natural Science Foundation of China
  • Research Grants Council of the Hong Kong SAR
  • InnoHK initiative, the Government of the HKSAR, Laboratory for AI-Powered Financial Technologies.

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)414
  • Downloads (Last 6 weeks)78
Reflects downloads up to 12 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2025)Privacy-preserved and Responsible Recommenders: From Conventional Defense to Federated Learning and BlockchainACM Computing Surveys10.1145/370898257:5(1-35)Online publication date: 9-Jan-2025
  • (2024)Federated Learning-Based Social Recommendation with Social Denoising2024 International Conference on New Trends in Computational Intelligence (NTCI)10.1109/NTCI64025.2024.10776360(331-335)Online publication date: 18-Oct-2024
  • (2024)Large Language Models Augmented Rating Prediction in Recommender SystemICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)10.1109/ICASSP48485.2024.10447514(7960-7964)Online publication date: 14-Apr-2024

View Options

Login options

Full Access

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Full Text

View this article in Full Text.

Full Text

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media