{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,6]],"date-time":"2026-02-06T04:19:04Z","timestamp":1770351544156,"version":"3.49.0"},"reference-count":61,"publisher":"Cambridge University Press (CUP)","issue":"11","license":[{"start":{"date-parts":[[2025,10,24]],"date-time":"2025-10-24T00:00:00Z","timestamp":1761264000000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/www.cambridge.org\/core\/terms"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Robotica"],"published-print":{"date-parts":[[2025,11]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>In the fields of meal-assisting robotics and human\u2013robot interaction (HRI), real-time and accurate mouth pose estimation is critical for ensuring interaction safety and improving user experience. The complexity arises from the diverse opening degrees of mouths, variations in orientation, and external factors such as lighting conditions and occlusions, which pose significant challenges for real-time and accurate posture estimation of mouths. In response to the above-mentioned issues, this paper proposes a novel method for point cloud fitting and posture estimation of mouth opening degrees (FP-MODs). The proposed method leverages both RGB and depth images captured from a single viewpoint, integrating geometric modeling with advanced point cloud processing techniques to achieve robust and accurate mouth posture estimation. The innovation of this work lies in the hypothesis that different states of mouth openings can be effectively described by distinct geometric shapes: closed mouths are modeled by spatial quadratic surfaces, half-open mouths by spatial ellipses, and fully open mouths by spatial circles. Then, based on these hypotheses, we developed algorithms for fitting geometric models to point clouds obtained from mouth regions, respectively. Specifically, for the closed mouth state, we employ an algorithm based on least squares optimization to fit a spatial quadratic surface to the point cloud data. For the half-open or fully open mouth states, we combine inverse projection methods with least squares fitting to model the contour as a spatial ellipse and circle, respectively. Finally, to evaluate the effectiveness of the proposed FP-MODs method, extensive actual experiments were conducted under varying conditions, including different orientations and various types of mouths. The results demonstrate that the proposed FP-MODs method achieves high accuracy and robustness. This study can provide a theoretical foundation and technical support for improving HRI and food delivery safety in the field of robotics.<\/jats:p>","DOI":"10.1017\/s0263574725102737","type":"journal-article","created":{"date-parts":[[2025,10,24]],"date-time":"2025-10-24T09:56:21Z","timestamp":1761299781000},"page":"4088-4115","source":"Crossref","is-referenced-by-count":0,"title":["Real-time mouth posture estimation for meal-assisting robots"],"prefix":"10.1017","volume":"43","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1549-4245","authenticated-orcid":false,"given":"Yuhe","family":"Fan","sequence":"first","affiliation":[{"name":"Harbin Engineering University"}]},{"given":"Lixun","family":"Zhang","sequence":"additional","affiliation":[{"name":"Harbin Engineering University"}]},{"given":"Canxing","family":"Zheng","sequence":"additional","affiliation":[{"name":"Weifang People\u2019s Hospital"}]},{"given":"Zhenhan","family":"Wang","sequence":"additional","affiliation":[{"name":"Harbin Engineering University"}]},{"given":"Zekun","family":"Yang","sequence":"additional","affiliation":[{"name":"Harbin Engineering University"}]},{"given":"Feng","family":"Xue","sequence":"additional","affiliation":[{"name":"Harbin Engineering University"}]},{"given":"Huaiyu","family":"Che","sequence":"additional","affiliation":[{"name":"Harbin Engineering University"}]},{"given":"Xingyuan","family":"Wang","sequence":"additional","affiliation":[{"name":"Harbin Engineering University"}]}],"member":"56","published-online":{"date-parts":[[2025,10,24]]},"reference":[{"key":"S0263574725102737_ref45","doi-asserted-by":"crossref","unstructured":"[45] Mo, S. and Miao, X. . Osgg-Net: One-Step Graph Generation Network for Unbiased Head Pose Estimation. In: Proc. 29th ACM Int. Conf. Multimedia (2021) pp. 2465\u20132473.","DOI":"10.1145\/3474085.3475417"},{"key":"S0263574725102737_ref7","unstructured":"[7] Suneel, B. , Ethan, K. , Yuxiao, C. , Siddhartha, S. , Tapomayukh, B. and Dorsa, S. , \u201cBalancing Efficiency and Comfort in Robot-Assisted Bite Transfer,\u201d In: Proc. 2022 IEEE Int. Conf. Robot. Autom., Shenyang, China (2022). arXiv:2111.11401."},{"key":"S0263574725102737_ref17","doi-asserted-by":"publisher","DOI":"10.1007\/s00521-021-06012-8"},{"key":"S0263574725102737_ref54","doi-asserted-by":"crossref","first-page":"122293","DOI":"10.1016\/j.eswa.2023.122293","article-title":"Real-time 6DoF full-range marker-less head pose estimation","volume":"239","author":"Redhwan","year":"2024","journal-title":"Expert Syst. Appl."},{"key":"S0263574725102737_ref29","doi-asserted-by":"publisher","DOI":"10.1017\/S0263574724001474"},{"key":"S0263574725102737_ref1","doi-asserted-by":"crossref","first-page":"1544","DOI":"10.1109\/LRA.2018.2801475","article-title":"A multimodal anomaly detector for robot-assisted feeding using an LSTM-based variational autoencoder","volume":"3","author":"Daehyung","year":"2018","journal-title":"IEEE Robot. Autom. Lett."},{"key":"S0263574725102737_ref18","doi-asserted-by":"crossref","first-page":"47795","DOI":"10.1109\/ACCESS.2021.3068045","article-title":"A comprehensive review of speech emotion recognition systems","volume":"9","author":"Taiba","year":"2021","journal-title":"IEEE Access"},{"key":"S0263574725102737_ref25","unstructured":"[25] Ethan, K. G. , Rajat, K. J. , Amal, N. , Ziang, L. , Tyler, S. and Haya, B. , \u201cAn Adaptable, Safe, and Portable Robot-Assisted Feeding System. In: Proc. 2024 ACM\/IEEE Int. Conf. Hum.-Robot Interact. March, Boulder, CO, USA (2024) pp. 11 (14."},{"key":"S0263574725102737_ref28","doi-asserted-by":"publisher","DOI":"10.1017\/S0263574721001508"},{"key":"S0263574725102737_ref32","doi-asserted-by":"crossref","unstructured":"[32] Whitehill, J. and Movellan, J. R. . A Discriminative Approach to Frame-By-Frame Head Pose tracking. In: Proc. IEEE Conf. Autom. Face Gesture Recognit. (2008) pp. 1\u20137.","DOI":"10.1109\/AFGR.2008.4813396"},{"key":"S0263574725102737_ref40","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2018.2885472"},{"key":"S0263574725102737_ref12","doi-asserted-by":"crossref","first-page":"2536","DOI":"10.1111\/1750-3841.16581","article-title":"Contact forces and motion behavior of non-Newtonian fluid\u2013solid food by coupled SPH\u2013FEM method","volume":"88","author":"Yuhe","year":"2023","journal-title":"J. Food Sci."},{"key":"S0263574725102737_ref3","doi-asserted-by":"crossref","first-page":"1850001","DOI":"10.1142\/S2529737618500016","article-title":"A learning from demonstration framework for implementation of a feeding task","volume":"2","author":"Nabil","year":"2018","journal-title":"Encycl. Semant. Comput. Robot. Intell."},{"key":"S0263574725102737_ref16","doi-asserted-by":"crossref","first-page":"269","DOI":"10.1007\/s00530-024-01472-z","article-title":"Instance segmentation of faces and mouth-opening degrees based on improved YOLOv8 method","volume":"30","author":"Yuhe","year":"2024","journal-title":"Multimedia Syst."},{"key":"S0263574725102737_ref58","doi-asserted-by":"publisher","DOI":"10.3390\/s19020428"},{"key":"S0263574725102737_ref47","doi-asserted-by":"crossref","unstructured":"[47] Xin, M. , Mo, S. and Lin, Y. . \u201cEva-gcn: Head Pose Estimation Based on Graph Convolutional Networks,\u201d In: Proc. IEEE Conf. Comput. Vis. Pattern Recognit (2021) pp. 1462\u20131471.","DOI":"10.1109\/CVPRW53098.2021.00162"},{"key":"S0263574725102737_ref43","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2015.2500221"},{"key":"S0263574725102737_ref56","doi-asserted-by":"crossref","first-page":"109","DOI":"10.1016\/j.cag.2018.05.001","article-title":"Automated outdoor depth-map generation and alignment","volume":"74","author":"Martin","year":"2018","journal-title":"Comput. Graph."},{"key":"S0263574725102737_ref4","doi-asserted-by":"crossref","first-page":"1","DOI":"10.3390\/robotics9010001","article-title":"Application of reinforcement learning to a robotic drinking assistant","volume":"9","author":"Tejas","year":"2019","journal-title":"Robotics"},{"key":"S0263574725102737_ref8","doi-asserted-by":"crossref","first-page":"611","DOI":"10.1007\/s10514-018-9733-6","article-title":"Multimodal anomaly detection for assistive robots","volume":"43","author":"Daehyung","year":"2019","journal-title":"Auton. Robot."},{"key":"S0263574725102737_ref61","doi-asserted-by":"publisher","DOI":"10.1111\/j.1467-8659.2007.01016.x"},{"key":"S0263574725102737_ref2","doi-asserted-by":"crossref","first-page":"4578","DOI":"10.3390\/s21134578","article-title":"A hybrid brain\u2013computer interface for real-life food assist robot control","volume":"21","author":"Jihyeon","year":"2021","journal-title":"Sensors"},{"key":"S0263574725102737_ref34","doi-asserted-by":"publisher","DOI":"10.1109\/TMM.2019.2903724"},{"key":"S0263574725102737_ref42","doi-asserted-by":"crossref","unstructured":"[42] Ghiass, R. S. , Arandjelovi, O. and Laurendeau, D. . Highly Accurate and Fully Automatic Head Pose Estimation from a Low Quality Consumer-Level RGB-d Sensor. In: Proc. Workshop Comput. Models Soc. Interact. Hum.-Comput.-Media Commun. (2015) pp. 25\u201334.","DOI":"10.1145\/2810397.2810401"},{"key":"S0263574725102737_ref22","doi-asserted-by":"crossref","unstructured":"[22] Wei, X. , Li, C. T. and Hu, Y. , \u201cRobust Face Recognition Under Varying Illumination and Occlusion Considering Structured Sparsity,\u201d In: Proc. 2012 Int. Conf. Digit. Image Comput. Tech. Appl., Fremantle, WA, Australia (2012) pp. 1\u20137.","DOI":"10.1109\/DICTA.2012.6411704"},{"key":"S0263574725102737_ref41","doi-asserted-by":"crossref","unstructured":"[41] Martin, M. , Van, F. and Stiefelhagen, R. . Real Time Head Model Creation and Head Pose Estimation on Consumer Depth Cameras. In: Proc. Int. Conf. 3D Vis. (2014) pp. 641\u2013648.","DOI":"10.1109\/3DV.2014.54"},{"key":"S0263574725102737_ref31","doi-asserted-by":"crossref","first-page":"112485","DOI":"10.1016\/j.jfoodeng.2025.112485","article-title":"Measuring posture and volume of meals for meal-assisting robotics","volume":"392","author":"Yuhe","year":"2025","journal-title":"J. Food Eng."},{"key":"S0263574725102737_ref9","doi-asserted-by":"crossref","first-page":"100","DOI":"10.1515\/arh-2022-0127","article-title":"Rheological thixotropy and pasting properties of food thickening gums orienting at improving food holding rate","volume":"32","author":"Yuhe","year":"2022","journal-title":"Appl. Rheol."},{"key":"S0263574725102737_ref39","doi-asserted-by":"crossref","unstructured":"[39] Ruiz, N. , Chong, E. and Rehg, J. M. . Fine-Grained Head Pose Estimation Without Keypoints. In: Proc. IEEE Conf. Comput. Vis. Pattern Recognit (2018) pp. 2074\u20132083.","DOI":"10.1109\/CVPRW.2018.00281"},{"key":"S0263574725102737_ref48","doi-asserted-by":"publisher","DOI":"10.1109\/TITS.2023.3275070"},{"key":"S0263574725102737_ref33","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2011.5995458"},{"key":"S0263574725102737_ref11","doi-asserted-by":"crossref","first-page":"111448","DOI":"10.1016\/j.jfoodeng.2023.111448","article-title":"Motion behavior of non-Newtonian fluid-solid interaction foods","volume":"347","author":"Yuhe","year":"2023","journal-title":"J. Food Eng."},{"key":"S0263574725102737_ref46","doi-asserted-by":"publisher","DOI":"10.1109\/3DV53792.2021.00055"},{"key":"S0263574725102737_ref27","doi-asserted-by":"publisher","DOI":"10.1017\/S0263574716000874"},{"key":"S0263574725102737_ref36","doi-asserted-by":"crossref","unstructured":"[36] Baltru\u0161aitis, T. , Robinson, P. and Morency, L. P. . 3d Constrained Local Model for Rigid and Non-rigid Facial Tracking. In: Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (2012) pp. 2610\u20132617.","DOI":"10.1109\/CVPR.2012.6247980"},{"key":"S0263574725102737_ref10","doi-asserted-by":"crossref","first-page":"2301","DOI":"10.1080\/10942912.2022.2133141","article-title":"Viscoelasticity and friction of solid foods measurement by simulating meal-assisting robot","volume":"25","author":"Yuhe","year":"2022","journal-title":"Int. J. Food Prop."},{"key":"S0263574725102737_ref21","doi-asserted-by":"crossref","first-page":"215516","DOI":"10.1109\/ACCESS.2020.3040906","article-title":"Lip reading sentences using deep learning with only visual cues","volume":"8","author":"Souheil","year":"2020","journal-title":"IEEE Access"},{"key":"S0263574725102737_ref53","unstructured":"[53] Zhou, H. , Jiang, F. and Lu, H. , \u201cA simple baseline for direct 2D multi-person head pose estimation with full-range angles (2023), arXiv preprint, arXiv:2302.01110."},{"key":"S0263574725102737_ref20","doi-asserted-by":"crossref","first-page":"104669","DOI":"10.1016\/j.engappai.2022.104669","article-title":"A comprehensive survey on 3D face recognition methods","volume":"110","author":"Menghan","year":"2022","journal-title":"Eng. Appl. Artif. Intell."},{"key":"S0263574725102737_ref15","doi-asserted-by":"crossref","first-page":"9257","DOI":"10.1007\/s11760-024-03543-0","article-title":"Real-time and accurate detection for face and mouth openings in meal-assisting robotics","volume":"18","author":"Yuhe","year":"2024","journal-title":"Signal, Image Video Process"},{"key":"S0263574725102737_ref38","doi-asserted-by":"publisher","DOI":"10.1016\/j.robot.2018.01.005"},{"key":"S0263574725102737_ref57","doi-asserted-by":"crossref","first-page":"106107","DOI":"10.1016\/j.compag.2021.106107","article-title":"Three-dimensional reconstruction of guava fruits and branches using instance segmentation and geometry analysis","volume":"184","author":"Guichao","year":"2021","journal-title":"Comput. Electron. Agric."},{"key":"S0263574725102737_ref59","doi-asserted-by":"publisher","DOI":"10.1016\/j.compeleceng.2022.107685"},{"key":"S0263574725102737_ref60","doi-asserted-by":"crossref","first-page":"112818","DOI":"10.1016\/j.jfoodeng.2025.112818","article-title":"Study on adaptive fuzzy force control based on food rheology properties","volume":"406","author":"Yuhe","year":"2026","journal-title":"J. Food Eng."},{"key":"S0263574725102737_ref37","doi-asserted-by":"crossref","unstructured":"[37] Ahn, B. , Park, J. and Kweon, I. S. . Real-Time Head Orientation From a Monocular Camera Using Deep Neural Network. In: Proc. Asian Conf. Comput. Vis. (2014) pp. 82\u201396.","DOI":"10.1007\/978-3-319-16811-1_6"},{"key":"S0263574725102737_ref23","doi-asserted-by":"publisher","DOI":"10.1016\/j.procs.2022.01.181"},{"key":"S0263574725102737_ref24","doi-asserted-by":"publisher","DOI":"10.1007\/s00371-022-02511-4"},{"key":"S0263574725102737_ref51","doi-asserted-by":"crossref","unstructured":"[51] Hempel, T. , Abdelrahman, A. A. and Al-Hamadi, A. , \u201c6D rotation representation for unconstrained head pose estimation,\u201d Proc, IEEE Int. Conf. Image Process 2022, 2496\u20132500 (2022).","DOI":"10.1109\/ICIP46576.2022.9897219"},{"key":"S0263574725102737_ref6","first-page":"1","article-title":"Robot-assisted feeding: A technical application that combines learning from demonstration and visual interaction","volume":"1","author":"Fei","year":"2020","journal-title":"Technol. Health Care"},{"key":"S0263574725102737_ref52","doi-asserted-by":"publisher","DOI":"10.1016\/j.patcog.2021.108210"},{"key":"S0263574725102737_ref26","doi-asserted-by":"publisher","DOI":"10.1017\/S026357472400198X"},{"key":"S0263574725102737_ref19","unstructured":"[19] Xuening, W. , Zhaopeng, Q. and Chongchong, Y. . Multi-stage Multi-modalities Fusion of Lip, Tongue and Acoustics Information for Speech Recognition. In: Proc. 6th AI Cloud Comput. Conf., Kyoto, Japan (2023) pp. 16\u201318."},{"key":"S0263574725102737_ref30","doi-asserted-by":"publisher","DOI":"10.1017\/S0263574723000644"},{"key":"S0263574725102737_ref50","doi-asserted-by":"crossref","unstructured":"[50] Cao, Z. , Chu, Z. , Liu, D. and Chen, Y. . \u201cA Vector-Based Representation to Enhance Head Pose Estimation,\u201d In: Proc. IEEE\/CVF Winter Conf. Appl. Comput. Vis. (2021) pp. 1188\u20131197.","DOI":"10.1109\/WACV48630.2021.00123"},{"key":"S0263574725102737_ref13","doi-asserted-by":"crossref","first-page":"111996","DOI":"10.1016\/j.jfoodeng.2024.111996","article-title":"Real-time and accurate meal detection for meal-assisting robots","volume":"371","author":"Yuhe","year":"2024","journal-title":"J. Food Eng."},{"key":"S0263574725102737_ref55","unstructured":"[55] Intel (2022) https:\/\/www.intelrealsense.com\/wp-content\/uploads\/2020\/06\/Intel-RealSense-D400-Series-Datasheet-June-2020.pdf (Online accessed October 2022), 2022."},{"key":"S0263574725102737_ref5","first-page":"1","article-title":"I-feed: A robotic platform of an assistive feeding robot for the disabled elderly population","volume":"2","author":"Fei","year":"2020","journal-title":"Technol. Health Care"},{"key":"S0263574725102737_ref49","doi-asserted-by":"publisher","DOI":"10.1109\/TIP.2023.3275535"},{"key":"S0263574725102737_ref14","doi-asserted-by":"crossref","first-page":"80","DOI":"10.1007\/s11554-024-01459-z","article-title":"Real\u2010time and accurate model of instance segmentation of foods","volume":"21","author":"Yuhe","year":"2024","journal-title":"J. Real-Time Image Process"},{"key":"S0263574725102737_ref35","doi-asserted-by":"crossref","unstructured":"[35] Saeed, A. and Al-Hamadi, A. . Boosted human head pose estimation using kinect camera. In: Proc. IEEE Int. Conf. Image Process (2015) pp. 1752\u20131756.","DOI":"10.1109\/ICIP.2015.7351101"},{"key":"S0263574725102737_ref44","doi-asserted-by":"crossref","first-page":"108210","DOI":"10.1016\/j.patcog.2021.108210","article-title":"Head pose estimation using deep neural networks and 3D point clouds","volume":"121","author":"Yuanquan","year":"2022","journal-title":"Pattern Recognit."}],"container-title":["Robotica"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.cambridge.org\/core\/services\/aop-cambridge-core\/content\/view\/S0263574725102737","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,2,3]],"date-time":"2026-02-03T22:25:25Z","timestamp":1770157525000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.cambridge.org\/core\/product\/identifier\/S0263574725102737\/type\/journal_article"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,10,24]]},"references-count":61,"journal-issue":{"issue":"11","published-print":{"date-parts":[[2025,11]]}},"alternative-id":["S0263574725102737"],"URL":"https:\/\/doi.org\/10.1017\/s0263574725102737","relation":{},"ISSN":["0263-5747","1469-8668"],"issn-type":[{"value":"0263-5747","type":"print"},{"value":"1469-8668","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,10,24]]}}}