skip to main content
research-article

Model-Agnostic Adaptive Testing for Intelligent Education Systems via Meta-learned Gradient Embeddings

Published: 06 November 2024 Publication History

Abstract

The field of education has undergone a significant revolution with the advent of intelligent systems and technology, which aim to personalize the learning experience, catering to the unique needs and abilities of individual learners. In this pursuit, a fundamental challenge is designing proper test for assessing the students’ cognitive status on knowledge and skills accurately and efficiently. One promising approach, referred to as Computerized Adaptive Testing (CAT), is to administrate computer-automated tests that alternately select the next item for each examinee and estimate their cognitive states given their responses to the selected items. Nevertheless, existing CAT systems suffer from inflexibility in item selection and ineffectiveness in cognitive state estimation, respectively. In this article, we propose a Model-Agnostic adaptive testing framework via Meta-leaned Gradient Embeddings, MAMGE for short, improving both item selection and cognitive state estimation simultaneously. For item selection, we design a Gradient Embedding-based Item Selector (GEIS) which incorporates the concept of gradient embeddings to represent items and selects the best ones that are both informative and representative. For cognitive state estimation, we propose a Meta-learned Cognitive State Estimator (MCSE) to automatically control the estimation process by learning to learn a proper initialization and dynamically inferred updates. Both MCSE and GEIS are inherently model-agnostic, and the two modules have an ingenious connection via meta-learned gradient embeddings. Finally, extensive experiments evaluate the effectiveness and flexibility of MAMGE.

References

[1]
Momin Abbas, Quan Xiao, Lisha Chen, Pin-Yu Chen, and Tianyi Chen. 2022. Sharp-maml: Sharpness-aware model-agnostic meta learning. In Proceedings of the International Conference on Machine Learning. PMLR, 10–32.
[2]
Jordan T. Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. 2019. Deep batch active learning by diverse, uncertain gradient lower bounds. In Proceedings of the International Conference on Learning Representations.
[3]
Philip Bachman, Alessandro Sordoni, and Adam Trischler. 2017. Learning algorithms for active learning. In Proceedings of the 34th International Conference on Machine Learning, Vol. 70. JMLR.org, 301–310.
[4]
Sungyong Baik, Myungsub Choi, Janghoon Choi, Heewon Kim, and Kyoung Mu Lee. 2020a. Meta-learning with adaptive hyperparameters. Advances in Neural Information Processing Systems 33 (2020), 20755–20765.
[5]
Sungyong Baik, Seokil Hong, and Kyoung Mu Lee. 2020b. Learning to forget for meta-learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2379–2387.
[6]
Maria-Florina Balcan, Andrei Broder, and Tong Zhang. 2007. Margin based active learning. In Proceedings of the International Conference on Computational Learning Theory. Springer, 35–50.
[7]
Md Abul Bashar and Richi Nayak. 2021. Active learning for effectively fine-tuning transfer learning to downstream task. ACM Transactions on Intelligent Systems and Technology (TIST) 12, 2 (2021), 1–24.
[8]
Haoyang Bi, Haiping Ma, Zhenya Huang, Yu Yin, Qi Liu, Enhong Chen, Yu Su, and Shijin Wang. 2020. Quality meets diversity: A model-agnostic framework for computerized adaptive testing. In Proceedings of the IEEE International Conference on Data Mining (ICDM). IEEE, 42–51.
[9]
Hua-Hua Chang. 2015. Psychometrics behind computerized adaptive testing. Psychometrika 80, 1 (2015), 1–20.
[10]
Hua-Hua Chang and Zhiliang Ying. 1996. A global information approach to computerized adaptive testing. Applied Psychological Measurement 20, 3 (1996), 213–229.
[11]
Seung W. Choi, Matthew W. Grady, and Barbara G. Dodd. 2011. A new stopping rule for computerized adaptive testing. Educational and Psychological Measurement 71, 1 (2011), 37–53.
[12]
Susan E. Embretson and Steven P. Reise. 2013. Item Response Theory. Psychology Press.
[13]
Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the International Conference on Machine Learning. PMLR, 1126–1135.
[14]
Yarin Gal, Riashat Islam, and Zoubin Ghahramani. 2017. Deep bayesian active learning with image data. In Proceedings of the International Conference on Machine Learning. PMLR, 1183–1192.
[15]
Weibo Gao, Qi Liu, Zhenya Huang, Yu Yin, Haoyang Bi, Mu-Chun Wang, Jianhui Ma, Shijin Wang, and Yu Su. 2021. RCD: Relation map driven cognitive diagnosis for intelligent education systems. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. 501–510.
[16]
Aritra Ghosh and Andrew Lan. 2021. BOBCAT: Bilevel optimization-based computerized adaptive testing. In Proceedings of the International Joint Conference on Artificial Intelligence.
[17]
Stephen M. Haley, Pengsheng Ni, Ronald K. Hambleton, Mary D. Slavin, and Alan M. Jette. 2006. Computer adaptive testing improved accuracy and precision of scores over random item selection in a physical functioning item bank. Journal of Clinical Epidemiology 59, 11 (2006), 1174–1182.
[18]
Yuting Hong, Shiwei Tong, Wei Huang, Yan Zhuang, Qi Liu, Enhong Chen, Xin Li, and Yuanjing He. 2023. Search-efficient computerized adaptive testing. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management. 773–782.
[19]
Giles Hooker, Matthew Finkelman, and Armin Schwartzman. 2009. Paradoxical results in multidimensional item response theory. Psychometrika 74, 3 (2009), 419–442.
[20]
Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. 2021. Meta-learning in neural networks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence 44, 9 (2021), 5149–5169.
[21]
H. Jiaji, C. Rewon, R. Vinay, L. Hairong, S. Sanjeev, and C. Adam. 2016. Active learning for speech recognition: The power of gradients. In Proceedings of the 30th Conference on Neural Information Processing Systems. NIPS, Barcelona, Spain, 1–5.
[22]
Weisen Jiang, James Kwok, and Yu Zhang. 2022. Subspace learning for effective meta-learning. In Proceedings of the International Conference on Machine Learning. PMLR, 10177–10194.
[23]
Gregory Koch, Richard Zemel, and Ruslan Salakhutdinov. 2015. Siamese neural networks for one-shot image recognition. In ICML Deep Learning Workshop, Vol. 2. Lille.
[24]
Ksenia Konyushkova, Raphael Sznitman, and Pascal Fua. 2017. Learning active learning from data. Advances in Neural Information Processing Systems 30 (2017), 4228–4238.
[25]
Elisabeth Lex and Markus Schedl. 2022. Psychology-informed recommender systems tutorial. In Proceedings of the 16th ACM Conference on Recommender Systems. 714–717.
[26]
Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. 2017. Meta-sgd: Learning to learn quickly for few-shot learning. arXiv:1707.09835.
[27]
Qi Liu, Zhenya Huang, Yu Yin, Enhong Chen, Hui Xiong, Yu Su, and Guoping Hu. 2019. Ekt: Exercise-aware knowledge tracing for student performance prediction. IEEE Transactions on Knowledge and Data Engineering 33, 1 (2019), 100–115.
[28]
Qi Liu, Runze Wu, Enhong Chen, Guandong Xu, Yu Su, Zhigang Chen, and Guoping Hu. 2018. Fuzzy cognitive diagnosis for modelling examinee performance. ACM Transactions on Intelligent Systems and Technology (TIST) 9, 4 (2018), 1–26.
[29]
Shuo Liu, Junhao Shen, Hong Qian, and Aimin Zhou. 2024. Inductive cognitive diagnosis for fast student learning in web-based online intelligent education systems. In Proceedings of the ACM Web Conference 2024.
[30]
Wenhe Liu, Xiaojun Chang, Ling Chen, Dinh Phung, Xiaoqin Zhang, Yi Yang, and Alexander G. Hauptmann. 2020. Pair-based uncertainty and diversity promoting early active learning for person re-identification. ACM Transactions on Intelligent Systems and Technology 11, 2 (2020), 1–15.
[31]
Frederic M. Lord. 1980. Applications of Item Response Theory to Practical Testing Problems. Routledge.
[32]
David Magis, Duanli Yan, and Alina A. Von Davier. 2017. Computerized Adaptive and Multistage Testing with R: Using Packages CATR and MSTR. Springer.
[33]
Andrew J. Martin and Goran Lazendic. 2018. Computer-adaptive testing: Implications for students’ achievement, motivation, engagement, and subjective test experience. Journal of Educational Psychology 110, 1 (2018), 27.
[34]
Craig N. Mills and Manfred Steffen. 2000. The GRE computer adaptive test: Operational issues. In: Van der Linden Wim J., Glas, Gees A. W. (Eds.), Computerized Adaptive Testing: Theory and Practice. Springer, 75–99.
[35]
Tsendsuren Munkhdalai and Hong Yu. 2017. Meta networks. In Proceedings of the International Conference on Machine Learning. PMLR, 2554–2563.
[36]
Darkhan Nurakhmetov. 2019. Reinforcement learning applied to adaptive classification testing. Theoretical and Practical Advances in Computer-Based Educational Measurement (2019), 325–336.
[37]
Martin Plajner and Jirí Vomlel. 2016. Student skill models in adaptive testing. In Probabilistic Graphical Models. JMLR: Workshop and Conference Proceedings vol 52. 403–414.
[38]
Lawrence M. Rudner. 2009. Implementing the graduate management admission test computerized adaptive test. In: Van der Linden Wim J., Glas Cees A. W. (Eds.), Elements of Adaptive Testing. Springer, 151–165.
[39]
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. 2016. Meta-learning with memory-augmented neural networks. In Proceedings of the International Conference on Machine Learning. PMLR, 1842–1850.
[40]
Ozan Sener and Silvio Savarese. 2017. Active learning for convolutional neural networks: A core-set approach. arXiv:1708.00489.
[41]
Burr Settles. 2011. From theories to queries: Active learning in practice. In Active Learning and Experimental Design Workshop in Conjunction with AISTATS 2010. JMLR Workshop and Conference Proceedings, 1–18.
[42]
Junhao Shen, Hong Qian, Wei Zhang, and Aimin Zhou. 2024b. Symbolic cognitive diagnosis via hybrid optimization for intelligent education systems. In Proceedings of the 38th AAAI Conference on Artificial Intelligence. 14928–14936.
[43]
Shuanghong Shen, Qi Liu, Zhenya Huang, Yonghe Zheng, Minghao Yin, Minjuan Wang, and Enhong Chen. 2024a. A survey of knowledge tracing: Models, variants, and applications. IEEE Transactions on Learning Technologies (2024), 1–22.
[44]
Jake Snell, Kevin Swersky, and Richard S. Zemel. 2017. Prototypical networks for few-shot learning. arXiv:1703.05175.
[45]
Yu Su, Zeyu Cheng, Jinze Wu, Yanmin Dong, Zhenya Huang, Le Wu, Enhong Chen, Shijin Wang, and Fei Xie. 2022. Graph-based cognitive diagnosis for intelligent tutoring systems. Knowledge-Based Systems 253 (2022), 109547.
[46]
Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip H. S. Torr, and Timothy M. Hospedales. 2018. Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1199–1208.
[47]
Daniel Ting and Eric Brochu. 2018. Optimal subsampling with influence functions. Advances in Neural Information Processing Systems 31, 3650–3659.
[48]
Wim J. van der Linden and Cees A. W. Glas. 2010. Elements of Adaptive Testing. Springer.
[49]
Jill-Jênn Vie, Fabrice Popineau, Éric Bruillard, and Yolaine Bourda. 2017. A review of recent advances in adaptive assessment. In: Peña-Ayala Alejandro (Ed.), Learning Analytics: Fundaments, Applications, and Trends. Springer, 113–142.
[50]
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. 2016. Matching networks for one shot learning. Advances in Neural Information Processing Systems 29 (2016), 3630–3638.
[51]
Howard Wainer, Neil J. Dorans, Ronald Flaugher, Bert F. Green, and Robert J. Mislevy. 2000. Computerized Adaptive Testing: A Primer. Routledge.
[52]
Chun Wang and Hua-Hua Chang. 2011. Item selection in multidimensional computerized adaptive testing—Gaining information from different angles. Psychometrika 76, 3 (2011), 363–384.
[53]
Fei Wang, Qi Liu, Enhong Chen, Zhenya Huang, Yu Yin, Shijin Wang, and Yu Su. 2023. NeuralCD: A general framework for cognitive diagnosis. IEEE Transactions on Knowledge and Data Engineering 35, 8 (2023), 8312–8327.
[54]
Hao Wang, Defu Lian, Hanghang Tong, Qi Liu, Zhenya Huang, and Enhong Chen. 2021. Hypersorec: Exploiting hyperbolic user and item representations with multiple aspects for social-aware recommendation. ACM Transactions on Information Systems (TOIS) 40, 2 (2021), 1–28.
[55]
Le Wu, Qi Liu, Enhong Chen, Nicholas Jing Yuan, Guangming Guo, and Xing Xie. 2016. Relevance meets coverage: A unified framework to generate diversified recommendations. ACM Transactions on Intelligent Systems and Technology (TIST) 7, 3 (2016), 1–30.
[56]
Lin Yang, Yizhe Zhang, Jianxu Chen, Siyuan Zhang, and Danny Z. Chen. 2017. Suggestive annotation: A deep active learning framework for biomedical image segmentation. In Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 399–407.
[57]
Lei Zhang, Wuji Zhang, Likang Wu, Ming He, and Hongke Zhao. 2023b. SHGCN: Socially enhanced heterogeneous graph convolutional network for multi-behavior prediction. ACM Transactions on the Web 18, 1 (2023), 1–27.
[58]
Ye Zhang, Matthew Lease, and Byron Wallace. 2017. Active discriminative text representation learning. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 31.
[59]
Zheng Zhang, Le Wu, Qi Liu, Jiayu Liu, Zhenya Huang, Yu Yin, Yan Zhuang, Weibo Gao, and Enhong Chen. 2023a. Understanding and improving fairness in cognitive diagnosis. SCIENCE CHINA Information Sciences Volume 67, 152106 (2024).
[60]
Chuang Zhao, Hongke Zhao, Xiaomeng Li, Ming He, Jiahui Wang, and Jianping Fan. 2023. Cross-domain recommendation via progressive structural alignment. IEEE Transactions on Knowledge and Data Engineering 36, 2401–2415.
[61]
Chanjin Zheng. 2015. Some practical item selection algorithms in cognitive diagnostic computerized adaptive testing–smart diagnosis for smart learning. Ph. D. Dissertation. University of Illinois at Urbana-Champaign.
[62]
Yan Zhuang, Qi Liu, Zhenya Huang, Zhi Li, Binbin Jin, Haoyang Bi, Enhong Chen, and Shijin Wang. 2022. A robust computerized adaptive testing approach in educational question retrieval. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval. 416–426.
[63]
Yingtian Zou, Fusheng Liu, and Qianxiao Li. 2021. Unraveling model-agnostic meta-learning via the adaptation learning rate. In Proceedings of the International Conference on Learning Representations.

Index Terms

  1. Model-Agnostic Adaptive Testing for Intelligent Education Systems via Meta-learned Gradient Embeddings

        Recommendations

        Comments

        Information & Contributors

        Information

        Published In

        cover image ACM Transactions on Intelligent Systems and Technology
        ACM Transactions on Intelligent Systems and Technology  Volume 15, Issue 5
        October 2024
        719 pages
        EISSN:2157-6912
        DOI:10.1145/3613688
        • Editor:
        • Huan Liu
        Issue’s Table of Contents

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        Published: 06 November 2024
        Online AM: 23 May 2024
        Accepted: 04 April 2024
        Revised: 16 March 2024
        Received: 14 August 2023
        Published in TIST Volume 15, Issue 5

        Check for updates

        Author Tags

        1. Adaptive testing
        2. intelligent tutoring system
        3. active learning
        4. meta-learning

        Qualifiers

        • Research-article

        Funding Sources

        • National Key Research and Development Program of China
        • National Natural Science Foundation of China
        • University Synergy Innovation Program of Anhui Province

        Contributors

        Other Metrics

        Bibliometrics & Citations

        Bibliometrics

        Article Metrics

        • 0
          Total Citations
        • 282
          Total Downloads
        • Downloads (Last 12 months)282
        • Downloads (Last 6 weeks)19
        Reflects downloads up to 12 Feb 2025

        Other Metrics

        Citations

        View Options

        Login options

        Full Access

        View options

        PDF

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader

        Full Text

        View this article in Full Text.

        Full Text

        Figures

        Tables

        Media

        Share

        Share

        Share this Publication link

        Share on social media