{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,31]],"date-time":"2026-03-31T16:29:16Z","timestamp":1774974556519,"version":"3.50.1"},"reference-count":29,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2023,2,4]],"date-time":"2023-02-04T00:00:00Z","timestamp":1675468800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"CAAI-Huawei MindSpore Open Fund","award":["CAAIXSJLJJ-2021-055A"],"award-info":[{"award-number":["CAAIXSJLJJ-2021-055A"]}]},{"name":"CAAI-Huawei MindSpore Open Fund","award":["2022JBZY021"],"award-info":[{"award-number":["2022JBZY021"]}]},{"DOI":"10.13039\/501100012226","name":"Fundamental Research Funds for the Central Universities","doi-asserted-by":"publisher","award":["CAAIXSJLJJ-2021-055A"],"award-info":[{"award-number":["CAAIXSJLJJ-2021-055A"]}],"id":[{"id":"10.13039\/501100012226","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100012226","name":"Fundamental Research Funds for the Central Universities","doi-asserted-by":"publisher","award":["2022JBZY021"],"award-info":[{"award-number":["2022JBZY021"]}],"id":[{"id":"10.13039\/501100012226","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>Currently, the most widely used protocol for the transportation layer of computer networks for reliable transportation is the Transmission Control Protocol (TCP). However, TCP has some problems such as high handshake delay, head-of-line (HOL) blocking, and so on. To solve these problems, Google proposed the Quick User Datagram Protocol Internet Connection (QUIC) protocol, which supports 0-1 round-trip time (RTT) handshake, a congestion control algorithm configuration in user mode. So far, the QUIC protocol has been integrated with traditional congestion control algorithms, which are not efficient in numerous scenarios. To solve this problem, we propose an efficient congestion control mechanism on the basis of deep reinforcement learning (DRL), i.e., proximal bandwidth-delay quick optimization (PBQ) for QUIC, which combines traditional bottleneck bandwidth and round-trip propagation time (BBR) with proximal policy optimization (PPO). In PBQ, the PPO agent outputs the congestion window (CWnd) and improves itself according to network state, and the BBR specifies the pacing rate of the client. Then, we apply the presented PBQ to QUIC and form a new version of QUIC, i.e., PBQ-enhanced QUIC. The experimental results show that the proposed PBQ-enhanced QUIC achieves much better performance in both throughput and RTT than existing popular versions of QUIC, such as QUIC with Cubic and QUIC with BBR.<\/jats:p>","DOI":"10.3390\/e25020294","type":"journal-article","created":{"date-parts":[[2023,2,6]],"date-time":"2023-02-06T02:57:32Z","timestamp":1675652252000},"page":"294","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":9,"title":["PBQ-Enhanced QUIC: QUIC with Deep Reinforcement Learning Congestion Control Mechanism"],"prefix":"10.3390","volume":"25","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8623-5262","authenticated-orcid":false,"given":"Zhifei","family":"Zhang","sequence":"first","affiliation":[{"name":"Engineering Research Center of Network Management Technology for High Speed Railway of Ministry of Education, School of Computer and Information Technology, Beijing Jiaotong University, Beijing 100044, China"},{"name":"Collaborative Innovation Center of Railway Traffic Safety, Beijing Jiaotong University, Beijing 100044, China"},{"name":"National Engineering Research Center of Advanced Network Technologies, Beijing Jiaotong University, Beijing 100044, China"}]},{"given":"Shuo","family":"Li","sequence":"additional","affiliation":[{"name":"Engineering Research Center of Network Management Technology for High Speed Railway of Ministry of Education, School of Computer and Information Technology, Beijing Jiaotong University, Beijing 100044, China"},{"name":"Collaborative Innovation Center of Railway Traffic Safety, Beijing Jiaotong University, Beijing 100044, China"},{"name":"National Engineering Research Center of Advanced Network Technologies, Beijing Jiaotong University, Beijing 100044, China"}]},{"given":"Yiyang","family":"Ge","sequence":"additional","affiliation":[{"name":"Engineering Research Center of Network Management Technology for High Speed Railway of Ministry of Education, School of Computer and Information Technology, Beijing Jiaotong University, Beijing 100044, China"},{"name":"Collaborative Innovation Center of Railway Traffic Safety, Beijing Jiaotong University, Beijing 100044, China"},{"name":"National Engineering Research Center of Advanced Network Technologies, Beijing Jiaotong University, Beijing 100044, China"}]},{"given":"Ge","family":"Xiong","sequence":"additional","affiliation":[{"name":"Collaborative Innovation Center of Railway Traffic Safety, Beijing Jiaotong University, Beijing 100044, China"},{"name":"China Software and Technical Service Co., Ltd., Beijing 100081, China"}]},{"given":"Yu","family":"Zhang","sequence":"additional","affiliation":[{"name":"Institute of Economics and Energy Supply and Demand, State Grid Energy Research Institute Co., Ltd., Beijing 102209, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9364-0207","authenticated-orcid":false,"given":"Ke","family":"Xiong","sequence":"additional","affiliation":[{"name":"Engineering Research Center of Network Management Technology for High Speed Railway of Ministry of Education, School of Computer and Information Technology, Beijing Jiaotong University, Beijing 100044, China"},{"name":"Collaborative Innovation Center of Railway Traffic Safety, Beijing Jiaotong University, Beijing 100044, China"},{"name":"National Engineering Research Center of Advanced Network Technologies, Beijing Jiaotong University, Beijing 100044, China"}]}],"member":"1968","published-online":{"date-parts":[[2023,2,4]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Langley, A., Riddoch, A., Wilk, A., Vicente, A., Krasic, C., Zhang, D., Yang, F., Kouranov, F., Swett, I., and Iyengar, J. (2017, January 21\u201325). The QUIC Transport Protocol: Design and Internet-Scale Deployment. Proceedings of the ACM SIGCOMM, Los Angeles, CA, USA.","DOI":"10.1145\/3098822.3098842"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"314","DOI":"10.1145\/52325.52356","article-title":"Congestion avoidance and control","volume":"18","author":"Jacobson","year":"1988","journal-title":"ACM SIGCOMM Comput. Commun. Rev."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Sander, C., R\u00fcth, J., Hohlfeld, O., and Wehrle, K. (2019, January 23). Deepcci: Deep learning-based passive congestion control identification. Proceedings of the 2019 Workshop on Network Meets AI & ML, Beijing, China.","DOI":"10.1145\/3341216.3342211"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Chen, X., Xu, S., Chen, X., Cao, S., Zhang, S., and Sun, Y. (2019, January 24\u201328). Passive TCP identification for wired and wireless networks: A long-short term memory approach. Proceedings of the 2019 15th International Wireless Communications & Mobile Computing Conference (IWCMC), Tangier, Morocco.","DOI":"10.1109\/IWCMC.2019.8766577"},{"key":"ref_5","first-page":"1","article-title":"Safe and nested subgame solving for imperfect-information games","volume":"30","author":"Brown","year":"2017","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_6","unstructured":"Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D., and Graepel, T. (2017). Mastering chess and shogi by self-play with a general reinforcement learning algorithm. arXiv."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Winstein, K., and Balakrishnan, H. (2013, January 27). TCP ex Machina: Computer-Generated Congestion Control. Proceedings of the ACM SIGCOMM 2013 Conference on SIGCOMM, Hong Kong, China.","DOI":"10.1145\/2486001.2486020"},{"key":"ref_8","unstructured":"Dong, M., Meng, T., Zarchy, D., Arslan, E., Gilad, Y., Godfrey, P.B., and Schapira, M. (2018, January 9\u201311). PCC Vivace: Online-Learning Congestion Control. Proceedings of the 15th Usenix Symposium on Networked Systems Design and Implementation (Nsdi\u201918) 2018, Renton, WA, USA."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"445","DOI":"10.1109\/TNSE.2018.2835758","article-title":"QTCP: Adaptive Congestion Control with Reinforcement Learning","volume":"6","author":"Li","year":"2019","journal-title":"IEEE Netw. Sci. Eng."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Abbasloo, S., Yen, C.-Y., and Chao, H.J. (2020, January 10\u201314). Classic Meets Modern: A Pragmatic Learning-Based Congestion Control for the Internet. Proceedings of the Annual conference of the ACM Special Interest Group on Data Communication on the Applications, Technologies, Architectures, and Protocols for Computer Communication, Online.","DOI":"10.1145\/3387514.3405892"},{"key":"ref_11","unstructured":"Jay, N., Rotman, N., Godfrey, B., Schapira, M., and Tamar, A. (2019, January 10\u201315). A Deep Reinforcement Learning Perspective on Internet Congestion Control. Proceedings of the 36th International Conference on Machine Learning, Long Beach, CA, USA."},{"key":"ref_12","unstructured":"Clemente, L., and Seemann, M. (2016, August 03). Quic-Go. Available online: https:\/\/github.com\/lucas-clemente\/quic-go."},{"key":"ref_13","unstructured":"Microsoft (2019, October 26). Msquic. Available online: https:\/\/github.com\/microsoft\/msquic."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"1210","DOI":"10.1049\/cmu2.12154","article-title":"Modified QUIC protocol with congestion control for improved network performance","volume":"15","author":"Kharat","year":"2021","journal-title":"IET Commun."},{"key":"ref_15","unstructured":"Cloudflare (2021, February 01). Quiche. Available online: https:\/\/github.com\/cloudflare\/quiche."},{"key":"ref_16","unstructured":"Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal Policy Optimization Algorithms. arXiv."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"20","DOI":"10.1145\/3012426.3022184","article-title":"BBR: Congestion-Based Congestion Control: Measuring bottleneck bandwidth and round-trip propagation time","volume":"14","author":"Cardwell","year":"2016","journal-title":"Queue"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"64","DOI":"10.1145\/1400097.1400105","article-title":"CUBIC: A New TCP-Friendly High-Speed TCP Variant","volume":"42","author":"Ha","year":"2008","journal-title":"ACM SIGOPS Oper. Syst. Rev."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Iyengar, J., and Thomson, M. (2021). RFC 9000 QUIC: A UDP-Based Multiplexed and Secure Transport. Omtermet Emgomeeromg Task Force, ACM Digital Library.","DOI":"10.17487\/RFC9000"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Yan, P., and Yu, N. (2022). The QQUIC Transport Protocol: Quantum-Assisted UDP Internet Connections. Entropy, 24.","DOI":"10.3390\/e24101488"},{"key":"ref_21","unstructured":"Kurose, J.F., and Ross, K.W. (2016). Computer Networking: A Top-Down Approach, Pearson FT Press. [7th ed.]."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Floyd, S., Kohler, E., and Padhye, J. (2006). Profile for Datagram Congestion Control Protocol (DCCP) Congestion Control ID 3: TCP-Friendly Rate Control (TFRC), RFC.","DOI":"10.17487\/rfc4342"},{"key":"ref_23","first-page":"527","article-title":"Network simulations with the ns-3 simulator","volume":"14","author":"Henderson","year":"2008","journal-title":"Sigcomm Demonstr."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Gawlowicz, P., and Zubow, A. (2019, January 25\u201329). ns-3 meets OpenAI Gym: The Playground for Machine Learning in Networking Research. Proceedings of the 22nd International Acm Conference on Modeling, Analysis and Simulation of Wireless and Mobile Systems, Miami Beach, FL, USA.","DOI":"10.1145\/3345768.3355908"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"De Biasio, A., Chiariotti, F., Polese, M., Zanella, A., and Zorzi, M. (2019, January 19). A QUIC Implementation for ns-3. Proceedings of the 2019 Workshop on NS-3, University of Florence, Florence, Italy.","DOI":"10.1145\/3321349.3321351"},{"key":"ref_26","unstructured":"Xu, L., Harfoush, K., and Rhee, I. (2004, January 7\u201311). Binary increase congestion control (BIC) for fast long-distance networks. Proceedings of the IEEE INFOCOM 2004, Hong Kong, China."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Rossi, D., Testa, C., Valenti, S., and Muscariello, L. (2010, January 2\u20135). LEDBAT: The new BitTorrent congestion control protocol. Proceedings of the 19th International Conference on Computer Communications and Networks, Zurich, Switzerland.","DOI":"10.1109\/ICCCN.2010.5560080"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Floyd, S., Henderson, T., and Gurtov, A. (2004). The NewReno Modification to TCP\u2019s Fast Recovery Algorithm, RFC.","DOI":"10.17487\/rfc3782"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"24","DOI":"10.1145\/190809.190317","article-title":"TCP Vegas: New techniques for congestion detection and avoidance","volume":"24","author":"Brakmo","year":"1994","journal-title":"ACM SIGCOMM Comput. Commun. Rev."}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/25\/2\/294\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T18:24:12Z","timestamp":1760120652000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/25\/2\/294"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,2,4]]},"references-count":29,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2023,2]]}},"alternative-id":["e25020294"],"URL":"https:\/\/doi.org\/10.3390\/e25020294","relation":{},"ISSN":["1099-4300"],"issn-type":[{"value":"1099-4300","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,2,4]]}}}