Play Mortal Kombat with the camera as your input device and your body/pose as a controller.

Video Tutorial: LRBY | YouTube

It converts given image of pose into keystroke (accorging to the given config). This is done broadly in 4 simple steps:

• OpenCV capture image if any motion is detected (via frame differencing)
• MediaPipe returns (x, y) coordinates of the each body part (if found)
• We use Mathematics to determine what pose is and which key that pose is associated with
• PyAutoGUI to input the keystroke

Move	Pose Image	Mathematics
UP		• ankles should be above knees
DOWN		• dist(hip->nose) : dist(hip->knee) > 7 : 10
LEFT		• (perpendicular of r_ankle->l_ankle) should be right of right_hip • slope(right_hip->ankle) for both ankles should be b/w -65° and -115°
RIGHT		• same as above but with left_hip
FRONT_PUNCH		• slope(left_shoulder->left_wirst) should be b/w -10.5° and 10.5° • angle(left_shoulder<-left_elbow->left_wirst) should be b/w 169.5° and 190.5° TLDR: arm should parallel to ground
BACK_PUNCH		• same as above but with right arm
FRONT_KICK		• slope(left_ankle->left_hip) should be b/w 45° and 135°
BACK_KICK		• same as above but with right leg
THROW		• wrist should be above elbow of same arm, they both should be above nose
TAG		• dist(wrists) : dist(elbows) < 3 : 10 • angle(left_elbow<-wirsts->right_elbow) should be b/w 169.5° and 190.5° TLDR: forearms should parallel to ground
BLOCK		• wrist should be above nose, they both should be above elbow of same arm

Lets start the standard procedure for python project setup.

• Clone the repository

$ git clone https://github.com/ra101/Pose2Input-MK9.git

• Create the virtualenv and activate it

$ cd Pose2Input-MK9
$ virtualenv venv
$ source ./venv/bin/activate # unix
$ .\venv\Scripts\activate.bat  # windows

• Install requirements

$ pip install -r requirements.txt

• copy content of .env.template into .env (one can use dump-env as well)

$ cat .env.template > .env

Env Help Guide:

One can simply run the application by this

$ python run.py

but for calibration, optional arguments are provided

Debug:

• Levels:
  • 0: Raw Video Footage
  • 1: 0 + FPS and Output Moves
  • 2: 1 + Virtual Exoskeleton of Body Parts Found
  • 3: 2 + Black Screen if no motion found
• If debug_level > 0 and no flag is selected, then log_flag is automatically set to True

Example of all flags being used:

$ python run.py --debug_level 3 --live_flag --log_flag