1266832

research-article20242024
PROXXX10.1177/10711813241266832Proceedings of the Human Factors and Ergonomics Society Annual MeetingLiu et al.

Poster Session

Proceedings of the Human Factors and

Artificial Intelligence in Virtual Reality for

Ergonomics Society Annual Meeting
2024, Vol. 68(1) 1333­–1338
Copyright © 2024 Human Factors
Blind and Low Vision Individuals: and Ergonomics Society
DOI: 10.1177/10711813241266832
https://doi.org/10.1177/10711813241266832

Literature Review journals.sagepub.com/home/pro

Tianhang Liu1 , Pooyan Fazli1, and Heejin Jeong1

Abstract
Virtual reality (VR) technologies have garnered substantial attention and adoption across various fields, including workspaces
and education. People with disabilities, who already face lower employment rates and fewer full-time opportunities, may be
further disadvantaged. Addressing this, recent advancements have focused on making VR inclusive, particularly for individuals
with visual impairments. With the rapid development of artificial intelligence (AI) technology, it has also become possible
to integrate AI technologies into VR systems to enhance accessibility for individuals with visual impairments. Despite
growing interest, there is a gap in comprehensive literature focusing on AI’s role in enabling blind and low vision individuals
to experience VR. This review aims to fill that void by providing an overview of current research, discussing benefits
and challenges, and identifying future opportunities. By synthesizing existing research, this study contributes insights for
researchers, developers, and practitioners working in the field of accessibility and assistive technology.

Keywords
artificial intelligence, virtual reality, blind and low vision, accessibility

Introduction vision struggle with small text, poor contrast, and complex
visuals. Head-mounted displays further complicate accessi-
Virtual reality (VR) technology allows users to immerse bility, restricting peripheral vision and posing difficulties for
themselves in diverse virtual experiences, with applications those with central visual field loss. Additionally, wearing
from entertainment to healthcare, education, and beyond. VR corrective lenses, which is common for individuals with
technologies have garnered substantial attention and adop- myopia, can interfere with using these VR headsets.
tion across various fields, including workspaces and educa- Over the past decade, there have been significant advance-
tion (Rupp et al., 2023). People with disabilities who already ments in developing technology to assist individuals with
face lower employment rates and fewer full-time opportuni- visual impairments to experience virtual reality (Collins
ties may be further disadvantaged given the predicted et al., 2023; Maidenbaum & Amedi, 2015; Zhao et al., 2018).
increase of VR in over 23 million jobs by 2030 (Rupp et al., In recent years, artificial intelligence (AI) technology has
2023). This calls for developing inclusive technology solu- experienced its blooming era. The intersection of AI and VR
tions that can cater to individuals with disabilities. Specially enables a plethora of applications such as training (particu-
for individuals with visual impairments, including blind and larly medical and military), gaming, robotics and autono-
low vision (BLV) individuals. mous cars, and advanced visualization (Hirzle et al., 2023;
Globally, in 2020, 1.1 billion people were living with vision Reiners et al., 2021). The development of the VR experience
loss. Among them, 43 million people are blind, 295 million itself can also be enhanced by using AI (Suzuki et al., 2023).
people have moderate to severe visual impairments, 258 mil- With the rapid development of artificial intelligence, it has
lion people have mild visual impairments, and 510 million also become possible to integrate AI technologies into virtual
people have near vision problems (Bourne et al., 2021). reality systems to enhance accessibility for individuals with
Naturally, it is no small matter to develop technologies and visual impairments. These AI technologies can provide
solutions that can improve accessibility and enhance the expe-
riences of individuals with blindness or visual impairments.
Blind and low vision individuals face significant chal- 1
Arizona State University, Mesa, AZ, USA
lenges in accessing VR experiences. VR technology, which
Corresponding Author:
relies heavily on visual components, is often inaccessible to Tianhang Liu, Arizona State University, 7418 E. Innovation Way South,
those with visual impairments. For completely blind indi- Mesa, AZ 85212, USA.
viduals, VR’s visual nature presents barriers. Those with low Email: tliu114@asu.edu
1334 Proceedings of the Human Factors and Ergonomics Society Annual Meeting 68(1)

innovative solutions and intuitive interactions that enable headset, immersive environment, virtual environ-
blind and low vision individuals to experience virtual, aug- ment, virtual space.
mented, and mixed reality (Fichten et al., 2023). •• BLV-related keywords: BVI, BLV, Blind, low vision,
Given the recent popularization of AI and VR, numerous visual impairment, vision loss, visual accessibility,
studies have been conducted to combine the two. Various lit- visual aids, assistive technology.
erature reviews have also been published to provide an over-
view of the current state of research at the intersection of AI When available, the search is performed with the following
and VR (Brzezinski & Krzeminska, 2023; Hirzle et al., 2023; search query using the Advanced Search function. An exam-
Reiners et al., 2021). These literature reviews have identified ple syntax used is (“AI” OR “artificial intelligence”) AND
several emerging trends, challenges, and application domains (“VR” OR “virtual reality”) AND (“BVI” OR “BLV”).
for AI and VR. Despite the growing research on applying AI
to assist blind and visually impaired individuals in experi-
encing VR, there is still a lack of a focused literature review
Selection Process
on the use of AI to enable these individuals to experience The review involves searching through academic databases
virtual reality. This review aims to fill this gap by analyzing such as IEEE Xplore, ACM Digital Library, PubMed, and
and summarizing the existing research on AI-enabled VR Google Scholar, to identify relevant research articles, confer-
experiences for blind and low vision individuals. The review ence papers, and other sources. Titles and abstracts of the
aims to address the following research question: “What retrieved articles were screened by one reviewer to assess
existing AI technologies and methods are being used to make their eligibility for inclusion in the review. Full-text articles
VR accessible to blind and low vision individuals?” were then assessed against predefined inclusion and exclu-
By conducting a thorough literature review, this paper sion criteria to determine their suitability for further
will contribute to the existing body of knowledge by synthe- analysis.
sizing and summarizing the research on AI-enabled VR tech- The inclusion criteria for selecting studies include
nologies for blind and low vision individuals, thereby research articles, conference papers, and technical reports
providing valuable insights for researchers, developers, and published in peer-reviewed journals or reputable conference
practitioners working in the field of accessibility and assis- proceedings that:
tive technology.
•• Utilization of virtual reality devices as the primary
medium.
Method •• Incorporation of artificial intelligence technologies.
This literature review was conducted following a methodology •• Explicit design targeting the blind and/or low vision
roughly aligned with the PRISMA 2020 (Preferred Reporting population.
Items for Systematic Reviews and Meta-Analyses) statement
(Page et al., 2021). While the review process adhered to key The exclusion criteria will include studies that:
principles outlined in the PRISMA 2020 statement, some adap-
tations were made to accommodate the scope and objectives of •• Non-English language publications.
the review. The search was done in May 2024. •• Inaccessibility of the study materials.
•• Absence of utilization of virtual reality devices.
•• Lack of integration of AI technologies.
Search Strategy •• Failure to focus on blind and/or low vision
The search strategy includes the identification of relevant individuals.
keywords and search terms related to using AI to enable
blind and low vision individuals to experience virtual reality. Additionally, citation chaining and snowballing techniques
Considering there are three major overlapping domains in were also used to identify additional relevant sources.
this topic—AI, XR, and BVI—the search terms will include
variations of these keywords. For each category, the follow-
Results
ing keywords will be used:
A total of 96 results were collected from the databases.
•• AI-related keywords: AI, Artificial intelligence, ML, Among them, seven irrelevant items were removed. During
machine learning, CV, computer vision, image rec- the first screening process, 59 of the total records were
ognition, NLP, natural language processing, deep excluded for being irrelevant. Of the remaining 30 reports,
learning, neural networks, pattern recognition, two were inaccessible, and two reports were excluded for not
generative. being centered around virtual reality, 22 reports were excluded
•• VR-related keywords: VR, virtual reality, HMD, head- for not using AI technology. A total of four studies were
mounted display, head-up display, head-worn display, included in this review (See Figure 1). The applications,
Liu et al. 1335

Figure 1. PRISMA flow diagram.

accessibility features, and AI technology used for enhancing requires a plethora of AI algorithms, including computer
VR accessibility for blind and low vision individuals were vision, natural language processing, and machine learning
extracted from each article. techniques. Polys and Wasi (2023) developed audio caption-
ing algorithms for enhancing accessibility in 3D contents on
the web. They also conducted a user study with 44 partici-
Audio Scene Description pants to evaluate the effectiveness of captioning algorithms
Of the four unique studies, two studies (Polys & Wasi, 2023; for search and summarize tasks. Their findings suggested
Zhao et al., 2019) focus on audio scene description, which that algorithms performed differently for different tasks and
1336 Proceedings of the Human Factors and Ergonomics Society Annual Meeting 68(1)

highlighted the significance of algorithm choice in enhanc- experience in using a walking cane, many expressed difficul-
ing accessibility for individuals with visual impairments in ties in navigating braille blocks. It’s worth noting that all
virtual environments. other included studies have tested, at least in part, with visu-
SeeingVR, developed by Zhao et al. (2019), focused on ally impaired individuals.
the development and evaluation of a set of 14 low vision
tools designed to enhance the accessibility of virtual reality
Synthesis
experiences for individuals with vision impairments. Among
the 14 tools, three of them deploy AI algorithms to achieve The existing AI technologies and methods used to assist
the desired functionality. The three tools are edge enhance- blind and low vision individuals in virtual reality include
ment for object detection, text-to-speech for reading textual Braille block recognition with convolutional neural net-
content, and audio scene description for scene recognition. works; obstacle avoidance and object selection through
Evaluation involving 11 participants with low vision demon- depth estimation and semantic edge detection; audio scene
strated that SeeingVR improved task completion rates and description using a combination of AI algorithms; and
overall enjoyment in VR environments. text-to-speech, which uses advanced natural language
processing techniques to convert written content into
audible speech.
Depth Estimation and Semantic Edge Detection Despite the limited number of articles reviewed, clear pat-
Similar to the edge enhancement tool by Zhao et al. (2019), one terns and insights have emerged. Among these techniques,
study enhances obstacle avoidance and object selection by edge enhancement and audio scene description are the most
combining depth estimation and semantic edge detection widely used. The audio scene description feature, like many
(Rasla & Beyeler, 2022), This study explored the relative others, mimics real-world counterpart AI solutions to pro-
importance of depth cues and semantic edges for indoor mobil- vide familiar and intuitive feedback for users. In contrast,
ity using simulated prosthetic vision in immersive virtual real- edge enhancement techniques might be more easily achiev-
ity. The researchers tested different scene simplification modes, able in virtual environments, than in the real world due to the
including DepthOnly, EdgesOnly, EdgesAndDepth, and controlled nature of VR settings. Techniques such as Braille
EdgesOrDepth. Tasks involved obstacle avoidance and object block recognition and audio scene description can assist both
selection in virtual environments to assess the effectiveness of blind and low vision individuals in navigating virtual reality,
these modes. Regarding obstacle avoidance, participants per- providing critical sensory feedback. On the other hand,
formed most successfully with the EdgesAndDepth mode, visual enhancement techniques like depth estimation and
achieving an average success rate of 89.8%. Additionally, par- semantic edge detection are particularly beneficial for those
ticipants were significantly faster using DepthOnly (17 s) com- with partial vision, helping them interpret and interact with
pared to EdgesOrDepth (22 s) and EdgesAndDepth (25 s). In their surroundings more effectively.
object selection, participants performed best with the
DepthOnly and EdgesOrDepth modes. Findings indicated the
significance of depth-based cues for obstacle avoidance and
Discussion
highlighted user preferences for certain scene simplification While the existing literature suggests that there is currently a
modes, emphasizing the importance of flexibility in visual lack of research specifically addressing the use of AI in
information presentation. enabling blind and low vision individuals to access virtual
reality experiences, there is a solid foundation of research on
using AI to enhance accessibility in other contexts. Some
Braille Blocks Recognition
studies have been conducted to investigate using artificial
The last study focuses on Braille block recognition using a intelligence technology to enhance various other aspects of
deep learning algorithm known as convolutional neural net- real-world experiences for blind and low vision individuals,
work (Kim, 2020). The study introduces a VR white cane such as video accessibility (Bodi et al., 2021), object recog-
equipped with a braille block recognition system that pro- nition (Gurari et al., 2018), navigation assistance (Kumar &
vides feedback through vibration and sound to help users Jain, 2022), and face recognition (Ibrahim & Saleh, 2009).
navigate virtual environments. By employing a decision- These studies demonstrate the potential of AI in improving
making model based on deep learning, the researchers devel- accessibility for blind and low vision individuals in various
oped an algorithm to identify Braille blocks through image domains, including virtual reality. Beyond that, AI technol-
inputs. The researchers conducted user surveys to assess sat- ogy like VizWiz (Gurari et al., 2018) has the potential to
isfaction and presence in the immersive VR environment and reduce the human labor that is typically required in modern
performance analysis experiments to evaluate the recogni- assistive technology. Some of the excluded studies also con-
tion of various braille block conditions. The results indicated ceptualized or mentioned the possibility of incorporating AI
high user satisfaction and effective braille block recognition. technology into current solutions (Aan et al., 2024; Collins
However, as they did not test with individuals with prior et al., 2023).
Liu et al. 1337

However, future researchers should keep in mind that AI References

may provide great utilities in many domains but is limited in Aan, H., Han, S., & Kim, K. (2024). A multiplayer vr showdown
many other domains (Smith & Smith, 2021). The limitations game for people with visual impairment. Human–Computer
of AI, particularly regarding disability, encompass chal- Interaction. Advance online publication. https://doi.org/10.10
lenges such as technical glitches, a lack of adaptability to 80/07370024.2024.2342961
diverse user requirements, as well as a lack of inclusivity. AI Bodi, A., Fazli, P., Ihorn, S., Siu, Y.-T., Scott, A. T., Narins, L., &
technologies sometimes disappoint with mistakes in function Yoon, I. (2021, May 8–13). Automated video description for
or interpretation, leading to frustrating experiences for users blind and low vision users. In Extended abstracts of the 2021 chi
with disabilities. There are also ethical dilemmas associated conference on human factors in computing systems, Yokohama,
Japan (pp. 1–7). https://doi.org/10.1145/3411763.3451810
with AI. Privacy, dependence, and equality issues are among
Bourne, R., Steinmetz, J. D., Flaxman, S., Briant, P. S., Taylor,
the many. AI systems could risk infringing on personal pri- H. R., Resnikoff, S., & Vos, T. (2021). Trends in prevalence
vacy if used in personal care or gathering sensitive data. of blindness and distance and near vision impairment over
Overdependence on AI could potentially weaken human 30 years: An analysis for the global burden of disease study.
autonomy and personal interactions. Moreover, the delega- The Lancet Global Health, 9(2), e130–e143. https://doi.
tion of decision-making to AI systems raises significant ethi- org/10.1016/S2214-109X(20)30425-3
cal questions. The AI’s ability to suggest choices or influence Brzezinski, M., & Krzeminska, I. (2023). The strategies for inno-
decisions could steer the user’s actions in a direction that vating with virtual reality and artificial intelligence: A litera-
may not be ethically correct. ture review. Technium: Romanian Journal of Applied Sciences
Another important challenge is the failure to involve dis- and Technology, 8, 72–83. https://doi.org/10.47577/technium.
abled people in the design process of AI technologies meant v8i.8671
Collins, J., Jung, C., Jang, Y., Montour, D., Won, A. S., & Azenkot,
for them. According to Smith and Smith (2021), the principle
S. (2023, October). “The guide has your back”: exploring how
of fairness in AI necessitates the contribution of disabled sighted guides can enhance accessibility in social virtual real-
people from the beginning stages of design, a practice that is ity for blind and low vision people. In Proceedings of the 25th
often not followed. To ensure inclusive AI technologies, it is international ACM SIGACCESS conference on computers and
crucial to involve disabled individuals in the design process accessibility (pp. 1–14). New York, NY, USA: Association for
to understand their unique needs and perspectives. Blind and Computing Machinery.
low vision individuals offer valuable firsthand insights and Fichten, C. S., Martiniello, N., Asuncion, J., Coughlan, T., & Havel,
perspectives during the design and testing phases of AI tech- A. (2023). Changing times: Emerging technologies for stu-
nologies, ensuring that these technologies are tailored to dents with disabilities in higher education. In J. W. Madaus,
their specific requirements. & L. L. Dukes, III (Eds.), Handbook of Higher Education and
Disability (pp. 131–148). Edward Elgar Publishing. https://doi.
org/10.4337/9781802204056.00019
Conclusion Gurari, D., Li, Q., Stangl, A. J., Guo, A., Lin, C., Grauman, K., &
Bigham, J. P. (2018, June 18–23). VizWiz grand challenge:
Enhancing accessibility to VR for blind and low vision indi- Answering visual questions from blind people. In Proceedings
viduals is important for their inclusion and equal participa- of the IEEE conference on computer vision and pattern recog-
tion in the virtual reality experience. The growing literature nition, Salt Lake City, UT, USA (pp. 3608–3617). https://doi.
investigating such topics shows a promising future. The inte- org/10.1109/CVPR.2018.00380
gration of AI technology with VR has the potential to signifi- Hirzle, T., Müller, F., Draxler, F., Schmitz, M., Knierim, P.,
cantly improve the accessibility and usability of VR for blind & Hornbæk, K. (2023, April). When XR and AI meet—A
and low vision individuals. More innovative methods and Scoping review on extended reality and artificial intelligence.
inclusive design practices are needed to ensure that AI tech- In Proceedings of the 2023 CHI conference on human fac-
nologies for VR are successfully developed, implemented, tors in computing systems (pp. 1–45). New York, NY, USA:
and utilized by blind and low vision individuals. Association for Computing Machinery.
Ibrahim, L. M., & Saleh, I. A. (2009). Face recognition using arti-
ficial intelligent techniques. ALRafidain Journal of Computer
Declaration of Conflicting Interests Sciences and Mathematics, 6(2), 211–227. https://doi.
The author(s) declared no potential conflicts of interest with respect org/10.33899/csmj.2009.163809
to the research, authorship, and/or publication of this article. Kim, J. (2020). VIVR: Presence of immersive interaction for visual
impairment virtual reality. IEEE Access: Practical Innovations,
Funding Open Solutions, 8, 196151–196159.
Kumar, N., & Jain, A. (2022). A deep learning based model to
The author(s) received no financial support for the research, author-
assist blind people in their navigation. Journal of Information
ship, and/or publication of this article.
Technology Education: Innovations in Practice, 21, 095–114.
https://doi.org/10.28945/5006
ORCID iD Maidenbaum, S., & Amedi, A. (2015, March). Non-visual virtual
Tianhang Liu https://orcid.org/0000-0002-2866-9430 interaction: Can Sensory Substitution generically increase the
1338 Proceedings of the Human Factors and Ergonomics Society Annual Meeting 68(1)

accessibility of Graphical virtual reality to the blind? In 2015 accessible: A discussion panel. Proceedings of the Human
3rd IEEE VR international workshop on virtual and augmented Factors and Ergonomics Society Annual Meeting, 67(1),
assistive technology (VAAT) (pp. 15–17). 1489–1494. https://doi.org/10.1177/21695067231192632
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Smith, P., & Smith, L. (2021). Artificial intelligence and disabil-
Hoffmann, T. C., & Mulrow, C. D.(2021). The prisma 2020 ity: Too much promise, yet too little substance? AI and Ethics,
statement: An updated guideline for reporting systematic 1(1), 81–86.
reviews. BMJ, 372, n71. Suzuki, R., Gonzalez-Franco, M., Sra, M., & Lindlbauer, D. (2023,
Polys, N., & Wasi, S. M. (2023). Increasing Web3D accessibility October). XR and AI: AI-Enabled Virtual, Augmented, and
with audio captioning. In Proceedings of the 28th international Mixed Reality. In Adjunct Proceedings of the 36th Annual ACM
ACM conference on 3D web technology. New York, NY, USA: Symposium on User Interface Software and Technology (pp. 1–
Association for Computing Machinery. 3). New York, NY, USA: Association for Computing Machinery.
Rasla, A., & Beyeler, M. (2022). The relative importance of depth cues Zhao, Y., Bennett, C. L., Benko, H., Cutrell, E., Holz, C., Morris,
and semantic edges for indoor mobility using simulated prosthetic M. R., & Sinclair, M. (2018, April). Enabling people with
vision in immersive virtual reality. In Proceedings of the 28th visual impairments to navigate virtual reality with a haptic
ACM symposium on virtual reality software and technology. New and auditory cane simulation. In Proceedings of the 2018 CHI
York, NY, USA: Association for Computing Machinery. conference on human factors in computing systems (pp. 1–14).
Reiners, D., Davahli, M. R., Karwowski, W., & Cruz-Neira, C. Montreal QC Canada: ACM.
(2021, September). The combination of artificial intelligence Zhao, Y., Cutrell, E., Holz, C., Morris, M. R., Ofek, E., & Wilson, A.
and extended reality: A systematic review. Frontiers in Virtual D. (2019, May). SeeingVR: A set of tools to make virtual real-
Reality, 2, 1933. https://doi.org/10.3389/frvir.2021.721933 ity more accessible to people with low vision. In Proceedings
Rupp, M. A., Gluck, A., Derby, J., Gable, T., Kelling, N., & Van of the 2019 CHI conference on human factors in computing
Ommen, C. (2023, September). Towards making XR 100% systems (pp. 1–14). Glasgow Scotland UK: ACM.