skip to main content
research-article

Federated Momentum Contrastive Clustering

Published: 18 June 2024 Publication History

Abstract

Self-supervised representation learning and deep clustering are mutually beneficial to learn high-quality representations and cluster data simultaneously in centralized settings. However, it is not always feasible to gather large amounts of data at a central entity, considering data privacy requirements and computational resources. Federated Learning (FL) has been developed successfully to aggregate a global model while training on distributed local data, respecting the data privacy of edge devices. However, most FL research effort focuses on supervised learning algorithms. A fully unsupervised federated clustering scheme has not been considered in the existing literature. We present federated momentum contrastive clustering (FedMCC), a generic federated clustering framework that can not only cluster data automatically but also extract discriminative representations training from distributed local data over multiple users. In FedMCC, we demonstrate a two-stage federated learning paradigm where the first stage aims to learn differentiable instance embeddings and the second stage accounts for clustering data automatically. The experimental results show that FedMCC not only achieves superior clustering performance but also outperforms several existing federated self-supervised methods for linear evaluation and semi-supervised learning tasks. Additionally, FedMCC can easily be adapted to ordinary centralized clustering through what we call momentum contrastive clustering (MCC). We show that MCC achieves state-of-the-art clustering accuracy results in certain datasets such as STL-10 and ImageNet-10. We also present a method to reduce the memory footprint of our clustering schemes.

References

[1]
Ahmed M. Abdelmoniem, Atal Narayan Sahu, Marco Canini, and Suhaib A. Fahmy. 2023. REFL: Resource-efficient federated learning. In Proceedings of the 18th European Conference on Computer Systems. 215–232.
[2]
Mohammed H. Alsharif, Jeong Kim, and Jin Hong Kim. 2017. Green and sustainable cellular base stations: An overview and future research directions. Energies 10, 5 (2017), 587.
[3]
Alp Berke Ardic, Hulya Seferoglu, Salim El Rouayheb, and Erdem Koyuncu. 2023. Random walking snakes for decentralized learning at edge networks. In Proceedings of the IEEE 29th International Symposium on Local and Metropolitan Area Networks (LANMAN’23). 1–6. DOI:
[4]
Bahman Bahmani, Benjamin Moseley, Andrea Vattani, Ravi Kumar, and Sergei Vassilvitskii. 2012. Scalable K-Means++. Retrieved from https://arxiv:1203.6402
[5]
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. 2021. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 9650–9660.
[6]
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning(Proceedings of Machine Learning Research, Vol. 119), Hal Daumé III and Aarti Singh (Eds.). PMLR, 1597–1607. Retrieved from https://proceedings.mlr.press/v119/chen20j.html
[7]
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. 2020. Improved baselines with momentum contrastive learning. Retrieved from https://arXiv:2003.04297
[8]
Xinlei Chen and Kaiming He. 2021. Exploring simple siamese representation learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’21). Computer Vision Foundation/IEEE, 15750–15758. Retrieved from https://openaccess.thecvf.com/content/CVPR2021/html/Chen_Exploring_Simple_Siamese_Representation_Learning_CVPR_2021_paper.html
[9]
Yihao Chen, Xianbiao Qi, Jianan Wang, and Lei Zhang. 2023. DisCo-CLIP: A distributed contrastive loss for memory efficient CLIP training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR’23). 22648–22657.
[10]
Inderjit S. Dhillon and Dharmendra S. Modha. 2002. A data-clustering algorithm on distributed memory multiprocessors. In Large-scale Parallel Data Mining. Springer, 245–260.
[11]
Chaowei Fang, Liang Wang, Dingwen Zhang, Jun Xu, Yixuan Yuan, and Junwei Han. 2022. Incremental cross-view mutual distillation for self-supervised medical CT synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 20677–20686.
[12]
Luyu Gao, Yunyi Zhang, Jiawei Han, and Jamie Callan. 2021. Scaling deep contrastive learning batch size under memory limited setup. Retrieved from https://arXiv:2101.06983
[13]
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. 2020. Bootstrap your Own Latent: A New Approach to Self-supervised Learning. Retrieved from https://arxiv:2006.07733
[14]
Xifeng Guo, En Zhu, Xinwang Liu, and Jianping Yin. 2018. Deep embedded clustering with data augmentation. In Proceedings of the Asian Conference on Machine Learning. PMLR, 550–565.
[15]
Alperen Görmez, Venkat R. Dasari, and Erdem Koyuncu. 2022. E2CM: Early exit via class means for efficient supervised and unsupervised learning. In Proceedings of the International Joint Conference on Neural Networks (IJCNN’22). 1–8. DOI:
[16]
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR’20).
[17]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’16). IEEE Computer Society, 770–778. DOI:
[18]
Li Huang, Andrew L. Shea, Huining Qian, Aditya Masurkar, Hao Deng, and Dianbo Liu. 2019. Patient clustering improves efficiency of federated machine learning to predict mortality and hospital stay time using distributed electronic medical records. J. Biomed. Inform. 99 (2019), 103291. DOI:
[19]
Zhizhong Huang, Jie Chen, Junping Zhang, and Hongming Shan. 2021. Exploring non-contrastive representation learning for deep clustering. Retrieved from https://arxiv.org/abs/2111.11821
[20]
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. 2020. SCAFFOLD: Stochastic controlled averaging for federated learning. In Proceedings of the 37th International Conference on Machine Learning(Proceedings of Machine Learning Research, Vol. 119), Hal Daumé III and Aarti Singh (Eds.). PMLR, 5132–5143. Retrieved from https://proceedings.mlr.press/v119/karimireddy20a.html
[21]
Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. Retrieved from https://arXiv:1412.6980
[22]
Erdem Koyuncu. 2023. Memorization capacity of neural networks with conditional computation. In Proceedings of the International Conference on Learning Representations.
[23]
Erdem Koyuncu. 2024. Centroidal clustering of noisy observations by using \(r\)th power distortion measures. IEEE Trans. Neural Netw. Learn. Syst. 35, 1 (2024), 1430–1438. DOI:
[24]
Tayfun Kucukyilmaz, University of Turkish Aeronautical Association et al. 2014. Parallel k-means algorithm for shared memory multiprocessors. J. Comput. Commun. 2, 11 (2014), 15.
[25]
Pengzhen Li, Erdem Koyuncu, and Hulya Seferoglu. 2023. Adaptive and resilient model-distributed inference in edge computing systems. IEEE Open J. Commun. Soc. 4 (2023), 1263–1273. DOI:
[26]
Pengzhen Li, Hulya Seferoglu, Venkat R. Dasari, and Erdem Koyuncu. 2021. Model-distributed DNN training for memory-constrained edge computing devices. In Proceedings of the IEEE International Symposium on Local and Metropolitan Area Networks (LANMAN’21). IEEE, 1–6.
[27]
Qinbin Li, Bingsheng He, and Dawn Song. 2021. Model-contrastive federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR ’21). 10713–10722.
[28]
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. 2020. Federated optimization in heterogeneous networks. In Proceedings of Machine Learning and Systems, I. Dhillon, D. Papailiopoulos, and V. Sze (Eds.), Vol. 2. 429–450. Retrieved from https://proceedings.mlsys.org/paper/2020/file/38af86134b65d0f10fe33d30dd76442e-Paper.pdf
[29]
Yunfan Li, Peng Hu, Jerry Zitao Liu, Dezhong Peng, Joey Tianyi Zhou, and Xi Peng. 2021. Contrastive clustering. In Proceedings of the 35th AAAI Conference on Artificial Intelligence (AAAI’21), the 33rd Conference on Innovative Applications of Artificial Intelligence (IAAI’21), and the 11th Symposium on Educational Advances in Artificial Intelligence (EAAI’21). AAAI Press, 8547–8555. Retrieved from https://ojs.aaai.org/index.php/AAAI/article/view/17037
[30]
James MacQueen et al. 1967. Some methods for classification and analysis of multivariate observations. In Proceedings of the 5th Berkeley Symposium on Mathematical Statistics and Probability. 281–297.
[31]
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. 2017. Communication-efficient learning of deep networks from decentralized data. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics(Proceedings of Machine Learning Research, Vol. 54), Aarti Singh and Jerry Zhu (Eds.). PMLR, 1273–1282. Retrieved from https://proceedings.mlr.press/v54/mcmahan17a.html
[32]
Hongyi Pan, Diaa Badawi, Runxuan Miao, Erdem Koyuncu, and Ahmet Enis Cetin. 2022. Multiplication-avoiding variant of power iteration with applications. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP’22). IEEE, 5608–5612.
[33]
Hieu Pham, Zihang Dai, Golnaz Ghiasi, Kenji Kawaguchi, Hanxiao Liu, Adams Wei Yu, Jiahui Yu, Yi-Ting Chen, Minh-Thang Luong, Yonghui Wu et al. 2023. Combined scaling for zero-shot transfer learning. Neurocomputing 555 (2023), 126658.
[34]
Hieu Pham, Zihang Dai, Golnaz Ghiasi, Kenji Kawaguchi, Hanxiao Liu, Adams Wei Yu, Jiahui Yu, Yi-Ting Chen, Minh-Thang Luong, Yonghui Wu et al. 2021. Combined scaling for open-vocabulary image classification. Retrieved from https://arxiv.org/abs/2111.10050
[35]
Trung Pham, Chaoning Zhang, Axi Niu, Kang Zhang, and Chang D. Yoo. 2022. On the pros and cons of momentum encoder in self-supervised visual representation learning. Retrieved from https://arXiv:2208.05744
[36]
Yuming Shen, Ziyi Shen, Menghan Wang, Jie Qin, Philip H. S. Torr, and Ling Shao. 2021. You never cluster alone. Retrieved from https://arxiv.org/abs/2106.01908
[37]
Shuai Wang and Tsung-Hui Chang. 2020. Federated clustering via matrix factorization models: From model averaging to gradient sharing. Retrieved from https://arxiv.org/abs/2002.04930
[38]
Yang Yang, Bo Wang, Dingwen Zhang, Yixuan Yuan, Qingsen Yan, Shijie Zhao, Zheng You, and Junwei Han. 2023. Self-supervised interactive embedding for one-shot organ segmentation. IEEE Trans. Biomed. Eng. 70, 10 (2023), 2799–2808.
[39]
Mang Ye, Xiuwen Fang, Bo Du, Pong C. Yuen, and Dacheng Tao. 2023. Heterogeneous federated learning: State-of-the-art and research challenges. Comput. Surveys 56, 3 (2023), 1–44.
[40]
Xiaohua Zhai, Avital Oliver, Alexander Kolesnikov, and Lucas Beyer. 2019. S4L: Self-supervised semi-supervised learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV’19).
[41]
Dingwen Zhang, Hao Li, Wenyuan Zeng, Chaowei Fang, Lechao Cheng, Ming-Ming Cheng, and Junwei Han. 2023. Weakly supervised semantic segmentation via alternate self-dual teaching. IEEE Trans. Image Process. (2023).
[42]
Fengda Zhang, Kun Kuang, Zhaoyang You, Tao Shen, Jun Xiao, Yin Zhang, Chao Wu, Yueting Zhuang, and Xiaolin Li. 2020. Federated unsupervised representation learning. Retrieved from https://arxiv.org/abs/2010.08982.
[43]
H. Zhong, J. Wu, C. Chen, J. Huang, M. Deng, L. Nie, Z. Lin, and X. Hua. 2021. Graph contrastive clustering. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV’21). IEEE Computer Society, Los Alamitos, CA, 9204–9213. DOI:
[44]
Sheng Zhou, Hongjia Xu, Zhuonan Zheng, Jiawei Chen, Jiajun Bu, Jia Wu, Xin Wang, Wenwu Zhu, Martin Ester et al. 2022. A comprehensive survey on deep clustering: Taxonomy, challenges, and future directions. Retrieved from arXiv:2206.07579.
[45]
Weiming Zhuang, Xin Gan, Yonggang Wen, Shuai Zhang, and Shuai Yi. 2021. Collaborative unsupervised visual representation learning from decentralized data. Retrieved from https://arxiv.org/abs/2108.06492
[46]
Weiming Zhuang, Yonggang Wen, and Shuai Zhang. 2022. Divergence-aware federated self-supervised learning. In Proceedings of the International Conference on Learning Representations. Retrieved from https://openreview.net/forum?id=oVE1z8NlNe

Cited By

View all
  • (2024)CLDP-pFedAvg: Safeguarding Client Data Privacy in Personalized Federated AveragingMathematics10.3390/math1222363012:22(3630)Online publication date: 20-Nov-2024
  • (2024)Information Theory in Emerging Wireless Communication Systems and NetworksEntropy10.3390/e2607054326:7(543)Online publication date: 26-Jun-2024
  • (2024)One-Shot Federated Clustering Based on Stable Distance RelationshipsIEEE Transactions on Industrial Informatics10.1109/TII.2024.343542020:11(13262-13272)Online publication date: Nov-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Intelligent Systems and Technology
ACM Transactions on Intelligent Systems and Technology  Volume 15, Issue 4
August 2024
563 pages
EISSN:2157-6912
DOI:10.1145/3613644
  • Editor:
  • Huan Liu
Issue’s Table of Contents

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 June 2024
Online AM: 26 March 2024
Accepted: 01 March 2024
Revised: 13 February 2024
Received: 27 November 2022
Published in TIST Volume 15, Issue 4

Check for updates

Author Tags

  1. Federated learning
  2. clustering
  3. contrastive learning
  4. unsupervised learning
  5. representation learning

Qualifiers

  • Research-article

Funding Sources

  • Army Research Lab (ARL)
  • Army Research Office (ARO)
  • National Science Foundation (NSF)

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)399
  • Downloads (Last 6 weeks)44
Reflects downloads up to 12 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)CLDP-pFedAvg: Safeguarding Client Data Privacy in Personalized Federated AveragingMathematics10.3390/math1222363012:22(3630)Online publication date: 20-Nov-2024
  • (2024)Information Theory in Emerging Wireless Communication Systems and NetworksEntropy10.3390/e2607054326:7(543)Online publication date: 26-Jun-2024
  • (2024)One-Shot Federated Clustering Based on Stable Distance RelationshipsIEEE Transactions on Industrial Informatics10.1109/TII.2024.343542020:11(13262-13272)Online publication date: Nov-2024
  • (2024)Rice cultivar clustering using federated K-means: focusing on advancing agriculture 4.0 applicationsGenetic Resources and Crop Evolution10.1007/s10722-024-02277-9Online publication date: 11-Dec-2024

View Options

Login options

Full Access

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Full Text

View this article in Full Text.

Full Text

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media