Jatin PDF
Jatin PDF
VIRTUAL MOUSE)
                  CLASS XII
                    Project Report
                           Submitted by:
                           Jatin Kaushal
        Board Roll No -___________________
Guided By
Mr. Vikas Tiwari
       CERTIFICATE
This is to certify that Jatin Kaushal of Class XII , student of
Allenhouse Public School , Rooma , Kanpur has successfully
completed their Artificial Intelligence Capstone Project on the topic
“ AI Virtual Mouse” under the guidance of “Mr. Vikas Tiwari”
during the year 2024-25.
I am satisfied with their initiative and efforts for the completion of
project file as a part of curriculum of CBSE Class XII Examination.
SIGNATURE                                      SIGNATURE
(PRINCIPAL)                               (SUBJECT TEACHER)
SIGNATURE
(EXAMINER)
ACKNOWLEDGEMENT
Date: __________
   AI PROJECT LOGBOOK
   Phase          Task        Planned      Planne   Plan     Actu       Actual     Actual      Who is      Remarks
                             start date    d end     ned     al        end        duratio   responsible
                                            date    dura     start     date       n
                                                     tion    date                 (hours,
                                                    (hou                          minutes
                                                     rs,                             )
                                                    minu
                                                     tes)
Preparing                    15-07-24     21-07-24 20-      15-07-24 21-07-24      20-30 hrs All Team
 for the                                            30                                      Members       Collabora
 project      Coursework
               readings                            hrs                                                    tive work
                  Rate         3/3
                yourselves
UnderstandinIdentify users   29-07-24     31-07-24 1 hr     29-07-24 30-07-24      1 hr     All Team      Collabora
 g the users                                                                                Members       tive work
                                                                                                            6
                 Interview      06-08-24   06-08-24 1 hr     06-08-24 06-08-24     1 hr   All Team
                 with user
                                                                                          Members
                    (2)
                 Interview      07-08-24   07-08-24 1hr      07-08-24 07-08-24     1hr     All Team
                   with                                                                    Members
                  user(3)
                 Interview      08-08-24   08-08-24 1hr      08-08-24 08-08-24     1hr     All Team
                   with                                                                    Members
                  user(4)
                 Complete       08-08-24   09-08-24   2hr    08-08-24 09-08-24     2 hr   All Team     Collabora
                 section 4                                                                Members      tive work
                   of the
                  Project
                 Logbook
                   Rate           3/3
                 yourselves
                   Team         09-08-24 10-08-24     2-3    09-08-24 10-08-24     3 hr   All Team     Collabora
Brainstorming     meeting                             hr                                  Members      tive work
                     to
                  generate
  Testing     Invite         02-11-24   02-11-24 2hr       02-11-24   02-11-24   2hr    All Team
  Creating    users    to                                                               Members
 the video    test your
               prototype
               Conduct       03-11-24              5hr     03-11-24   04-11-24   5hr    All Team
                                        04-11-24                                        Members
               testing
              with users
              Complete       05-11-24   05-11-24 4hr       05-11-24   05-11-24   4hr    All Team
              section 9                                                                 Members
                of the
               Project
              Logbook
                Rate           3/3
              yourselves
                Team         08-11-24   08-11-24 2hr       08-11-24   08-11-24   3hr    All Team
              meeting to                                                                Members
                discuss
                 video
               creation
                                                                                                      8
              Write         09-11-24   09-11-24 2hr     09-11-24   09-11-24    2hr    All Team
              your                                                                    Members
              script
              Film          09-11-24   09-11-24 4hr     09-11-24   09-11-24    3hr    All Team    Collabora
                your                                                                  Members     tive work
                video
2.3Communications plan
                                                                                                    9
    Who wasn’t able to attend: None
    Purpose of meeting: Research on topic, discussion of ideas and division of roles
    Items discussed:
        1.Define the purpose and objective of the AI virtual mouse.
        2.Determine specific use cases for individual with disabilities.
    Things to do (what, by whom, by when):
       1.Assign roles, oversee the entire project, set the timelines and track progress.(Sanya)
       2.Investigate and evaluate eye tracking technologies at AI algorithms .(Sukhdeep)
 2.4.1 Team meeting (Meeting 2)
  Date of meeting:22 July 24
  Who attended: Sanya,Nanshi,Sukhdeep,Jatin,Arnav and Manali
  Who wasn’t able to attend: None
  Purpose of meeting: To discuss solutions for the problem
  Items discussed: An Computer Vision powered image model responsible
      for eye tracking.
    Things to do: Learn and research on best AI powered framework that
      would help us make our model
 2.4.2 Team Meeting(Meeting 3)
    Date of meeting: 12 August 24
    Who attended: Sanya,Nanshi,Sukhdeep,Jatin,Arnav and Manali
    Who wasn’t able to attend: None
    Purpose of meeting: To design the solution
    Items discussed: Organizing and listing all the requirements required for
       construction and designing of model and decided to opt for image based
       recognition approach.
    Things to do: Start with data preparation and collection
                                                                                   11
                   3. PROBLEM DEFINITION
   3.1 List important local issues faced by your school or community ?
➢ The important local issue faced by our society is for disabled people to use
  technology and the people who are unable to use their hands.
                                                                                 12
                  4.THE USERS
    4.1 Who are the users and how they affected by the problem?
➢ The users are individuals with mobility impairments who cannot use
  traditional input devices like a mouse or keyboard. They are affected by
  limited access to technology, making it difficult to communicate, work, or
  perform daily tasks independently.
   4.2 What have you actually observed about the users and how the problem
   affects them?
➢ We have observed that users with mobility impairments face challenges in
  interacting with technology due to their inability to use traditional input
  devices. This limits their independence, reduces opportunities for education
  and employment, and hinders their ability to fully utilize digital tools.
4.3 Record your interview questions here as well as responses from users.
 Interviewer: Can you explain how eye tracking works in your AI Virtual Mouse
   Project?
➢ User: It uses computer vision (OpenCV, MediaPipe) to detect and track eye
   Movements, translating gaze positions into screen coordinates to
   control the Mouse pointer.
 Interviewer: What libraries or tools did you use to implement this AI Virtual
   Mouse?
➢ User: I used OpenCV for image processing, MediaPipe for facial landmarks,
   Pynput for stimulating mouse actions, and NumPy for mathematical
   calculations.
 Interviewer: What challenges did you face in implementing gaze tracking and
   how did you overcome it?
➢ User: Lightning conditions and head movements were challenging. I improved
   accuracy by refining facial landmark detection and using techniques to stabilize
   gaze-to screen mapping.
 Interviewer: How does it handle click events?
➢ User: Eye blinks are used for click detection.
                                                                                 13
    4.4 Empathy Map
               SAYS                                         THINKS
      It’s amazing that I can control my         Will it be tiring for my eyes
       computer just with my eyes.                  to use this for long? How
      It’s a bit challenging to control           can I improve my accuracy
       precise actions like dragging and            and speed with practice?
       dropping.
      I wish it could be more accurate in         How would it handle rapid
       low light conditions.                         movements or sudden
                                                     changes in gaze?
               DOES                                          FEELS
      Does the system to move the cursor          Feels like they have more
        using eye gaze.                               control especially with limited
      Calibrates the system regularly for            mobility.
        improved accuracy.                         Interested in the technology but
      May take breaks to avoid eye strain            concerned about its
        or fatigue.                                   effectiveness in real life use.
                                                    Eager to try an innovative way
                                                     of interacting with technology.
    4.5 What are the usual steps that users currently take related to the
    problem and where are the difficulties?
   Voice Control: Often imprecise and struggles in noisy environments.
   External Switches: Expensive, require setup, and can be physically
   challenging.
    Specialized Hardware: High cost and limited availability make it inaccessible
    for many users.
    General Difficulties: Limited accuracy, affordability, and accessibility.
                                                                                     14
                    5.BRAINSTORMING
              5.1 Ideas
 AI Idea #1      Use AI to accurately track eye movements and translate them into precise
                  cursor control, enabling hands-free interaction.
                  AI-driven speech recognition to assist users in controlling devices through
 AI Idea #2     
                  voice commands.
                  Machine learning to interpret specific head or facial gestures as input for device
 AI Idea #3     
                  interaction.
                  AI to adapt the gaze tracking system to individual user needs, enhancing
 AI Idea #4      accuracy and comfort.
                  AI-powered guidance and tutorials for users to learn how to use assistive
 AI Idea #5      technologies effectively.
    High value to users, easy to create             High value to users, hard to create
    ➢ Speech-to-Text                              ➢ Gaze Tracking
    Low value to users, easy to create             Low value to users, hard to create
    ➢ Accessibility Tutorials                     ➢ Gesture Recognition
   Easy                                                                           Hard
                                     EASE OF DEVELOPMENT
   5.6   Based on the priority grid, which AI solution is the best fit for your
         users and for your team to create and implement?
➢ Our approach is to create a gaze tracking model that we have trained using custom
  computer vision techniques to track eye movements. By leveraging MediaPipe and
  Python, we built the entire system from scratch, enabling hands-free interaction
  with technology. This approach allows individuals with mobility impairments to
  control their devices more efficiently, improving accessibility. We developed and
  trained the model to accurately track and translate gaze into cursor movement,
  ensuring a seamless user experience.
                                                                                           15
                                    6.DESIGN
6.1 What are the steps that users will now do using your AI solution to address the
problem?
  ➢ Calibration: Users will first calibrate the system by focusing on specific
     points on the screen to align the gaze tracker with their eye movements.
  ➢ Eye Tracking: Once calibrated, the system will continuously track the user’s
     eye movements in real-time.
  ➢ Cursor Control: Users will move the cursor on the screen by looking at
     different areas, with the system translating gaze direction into precise cursor
     movement.
  ➢ Interaction: Users can interact with their device by gazing at buttons, icons, or
     text, allowing for hands-free clicking and navigation.
                                                                                 16
                             7. DATA
                                                                                        17
                      8. PROTOTYPE
                                                                                 18
                                  9. TESTING
9.2 List your observations of your users as they tested your solution.
  ➢ Upon testing, we found that the model performed as expected, with an
     accuracy of 95%. The accuracy and F1 score for our model were 91% and
     0.8, respectively, demonstrating strong performance in gaze tracking and
     cursor control.
9.4 Refining the prototype: Based on user testing, what needs to be acted on
now so that the prototype can be used?
 ➢ User testing made us realize that background color is affecting model’s
 decision so we have decided to place the material on white background
 while using the model for image classification.
9.5 What improvements can be made later?
 ➢ Upon testing we found that the model was not fully predictive about some
   plastic material which are similar to look at. Upon discussing it with the
   team we have decided to update our training set with more diverse images
   especially images taken from usage by real life people.
                                                                                             19
                10.TEAM COLLABORATION
10.1 How did you actively work with others in your team and with stakeholders?
➢     We collaborated with our stakeholders by investigating the troubles they face
      on a daily basis and understanding them to help us make a project that aims
      to solve their troubles with full efficiency. Their inputs guided us throughout
      our journey.
➢     We as co-members created a group chat to stay in touch and communicate
      with each other efficiently at any time of the day comfortably. We made sure
      to have open ears for each others’ suggestions and did not let any member's
      efforts be unrecognized.
➢     We organized online meetings amongst the group members for
      brainstorming ideas.
➢     Sessions with mentors and stakeholders to test the efficiency of the
      prototype.
                                                                                20
11.INDIVIDUAL LEARNING REFLECTIONS
                                                                                21
                      12. VIDEO LINK
                                                                     22
                                                    Appendix
             Recommended Assessment Rubric (for Teachers)
                                  LOGBOOK AND VIDEO CONTENT
 Problem          A local problem which has not         A local problem which has not been       A local problem is
 definition        been fully solved before is           fully solved before is described.           described
                     explained in detail with
                      supporting research.
 The Users      Understanding of the user group is Understanding of the user group The user group is described
                evidenced by completion of all of   is evidenced by completion of   but it is unclear how they
                 the steps in Section 4 The Users   most of the steps in Section 4 are affected by the problem.
                   and thorough investigation.               The Users.
Brainstorming      A brainstorming session was         A brainstorming session was      A brainstorming session was
                conducted using creative and critical conducted using creative and      conducted. A solution was
                 thinking. A compelling solution      critical thinking. A solution was          selected.
                   was selected with supporting           selected with supporting
                    arguments from Section 5               arguments in Section 5
                         Brainstorming.                        Brainstorming.
                 The use of AI is a good fit for the      The use of AI is a good fit for    The use of AI is a good fit for
                 solution. The new user experience        the solution and there is some          the solution.
                is clearly documented showing how         documentation about how it
                users will be better served than they       meets the needs of users.
                             are today.
    Data           Relevant data to train the AI          Relevant data to train the AI      Relevant data to train the AI
                  model have been identified as           model have been identified as      model have been identified
                    well as how the data will be           well as how the data will be      as well as how the data will
                   sourced or collected. There is         sourced or collected. There is      be sourced or collected.
                    evidence that the dataset is           evidence that the dataset is
                  balanced, and that safety and                     balanced.
                  privacy have been considered.
                  A prototype for the solution has
  Prototype                                              A prototype for the solution has     A concept for a prototype
                  been created and successfully         been created and trained.             shows how the AI model
                      trained to meet users’                                                        will work
                          requirements.
   Testing      A prototype has been tested with a    A prototype has been tested      A concept for a prototype
                fair representation of users and all  with users and improvements     shows how it will be tested.
                  tasks in Section 9 Testing have     have been identified to meet
                          been completed.                  user requirements.
                 Effective team collaboration and    Team collaboration among peers
    Team                                                                            There is some evidence of team
collaboration   communication among peers and          and stakeholders is clearly  interactions among peers and
                stakeholders is clearly documented documented in Section 10 Team             stakeholders.
                in Section 10 Team collaboration.            collaboration.
 Individual       Each team member presents a                Each team presents an             Some team members
  learning      reflective and insightful account of         account of their learning      present an account of their
                their learning during the project.             during the project.          learning during the project.
                                                                                                                                23
                             VIDEO PRESENTATION
                                                                                     Points Given
                                                                                        3 – excellent
                                                                                       2 – very good
                                 Criteria                                             1 – satisfactory
  Accurate       The video presents accurate science and technology and uses
  language                      appropriate language.
 Sound and
                  The video demonstrates good sound and image quality.
image quality
Total points
24