Abstract
1 Real-Time Clothing Display
The proposed system allows users to view themselves wearing
virtual clothing in real-time, based on their body pose.
2 Technology Innovation
By combining deep learning models and computer vision algorithms ,
the system achieves high accuracy and speed, offering a seamless
interactive experience suitable for online retail platforms.
3 Enhanced Digital Shopping Experience
User feedback indicates that the system significantly improves the
overall digital shopping experience, making it a powerful tool for
retailers looking to leverage augmented reality in fashion.
preencoded.png 2
Table of Contents
1 ABSTRACT
2 INTRODUCTION
3 LITERATURE SURVEY
4 PROPOSED METHODOLOGY
5 POSE ESTIMATION MODEL
6 KALMAN FILTERING
7 COMMUNICATION METHODS
8 VIRTUAL CHARACTER DRIVING PROCESS
9 RESULTS
10 FUTURE ENHACEMENTS
11 CONCLUSIONS
12 REFERENCES
preencoded.png
3
Introduction
Human Pose Estimation
Human pose estimation in computer vision involves inferring body pose
information (joint positions, angles, movements) from images/videos,
with applications in areas like human-computer interaction, virtual
reality, and motion analysis.
Advancements
The rise of deep learning and large-scale datasets (e.g., MPII, COCO,
Human3.6M) has propelled research, particularly following the success
of CNNs like AlexNet in image recognition.
2D vs. 3D Pose Estimation
2D pose estimation detects joint positions in 2D images using techniques like
heatmap detection, facing challenges such as occlusion and lighting; while 3D
pose estimation determines spatial coordinates in 3D space, requiring depth
recovery and addressing complexities like viewpoint changes and depth
uncertainty. preencoded.png
4
Introduction(contd)
1 Modules
This chapter outlines the five main modules of the virtual
character interaction system: Pose Estimation Method, Kalman
Filtering, Communication, Virtual Character Driving, and Clothing
Swapping System.
2 Challenges
Challenges are occlusion, lighting changes, depth
uncertainty, specialized equipment requirements for 3D
data acquisition.
3 Real-Time Performance
The advancement of deep learning, particularly CNN has been
crucial in improving the accuracy and efficiency of human pose
estimation, enabling more sophisticated models that can handle
complex scenarios and multiple subjects simultaneously.
preencoded.png
5 5
Literature Survey
1.Vision Based Human Pose Estimation for Virtual Cloth Fitting(Sourav Saha,Pritha Ganguly)
This approach uses computer vision techniques to accurately estimate human body poses,
enabling real-time, realistic virtual clothing try-ons, enhancing online shopping experiences
while reducing return rates and improving customer satisfaction.
2.VNect: Real-time 3D Human Pose Estimation with a Single RGB Camera(Dushyant Mehta,Srinath
Sridhar)
Explores the VNect framework, focusing on its strengths in 3D pose estimation and how it can
enhance virtual garment fitting accuracy.
3.Human Pose Estimation Using MediaPipe Pose and Optimization Method Based on a Humanoid
Model(Jong Wook Kim,Jin Young Choi)
This approach integrates MediaPipe Pose for real-time human pose estimation with an
optimization method based on a humanoid model, enhancing the accuracy of joint tracking and
enabling more realistic virtual interactions and applications, such as virtual fitting and animation.
preencoded.png
6
Proposed Methodology
Pose Estimation Model
• MediaPipe is used to track the user's body keypoints in
real-time, providing the necessary data for virtual clothing
draping.
• Vnect addresses spatial ambiguity in 3D pose estimation by
jointly training 2D and 3D poses.
Kalman Filtering
Kalman filtering is applied to the keypoint data to stabilize
the tracking, ensuring a smooth and seamless virtual
clothing experience.
Unity3D Integration
The stabilized pose data is integrated into a Unity3D-based
platform, which handles the rendering of virtual clothing
assets aligned with the user's body.
preencoded.png
7
1.POSE ESTIMATION MODULE
MediaPipe
• Optimized for real-time performance with low computational
resources.
• Supports hardware acceleration (GPU/TPU) for faster inference.
• Modular design reduces computational overhead.
VNect
• Addresses spatial ambiguity in 3D pose estimation.
• Jointly trains 2D and 3D poses for accurate 3D information extraction.
MediaPipe Workflow:
• Input: Obtain real-time video stream.
• Detection: Extract human bounding box.
• Keypoint Detection: Use BlazePose to detect 33 key points and
estimate depth.
• Post-Processing: Apply filtering and pose reorientation.
• Visualization: Output a 3D skeleton overlay in the video.
preencoded.png
8
INTEFERNCE
PIPELINE
preencoded.png
9
2.KALMAN FILTERING MODULE
• The Kalman Filter estimates state variables in dynamic systems, providing optimal estimates by
combining a dynamic model and observational data.
• Used for estimating and tracking keypoint positions and velocities.
• Enhances accuracy and stability of pose detection by combining motion models with detection data.
preencoded.png
10
3.COMMUNICATION METHODS
To transmit human pose estimation data processed
through Kalman filtering from the server to the client in
real-time, socket communication with the UDP protocol is
utilized.
Implementation Steps
1.Socket Setup:
• Establish a UDP socket with an IP address and port.
• Transmit data as UDP datagrams.
2.Sending and Receiving Keypoint Data:
• In Python: Create a UDP socket to send keypoint data.
• In Unity: Set up a UDP client to receive and parse the
data.
Adjust from 33 MediaPipe keypoints to 16 keypoints for
Unity3D by introducing midpoints (e.g., between Hips and
Neck).
preencoded.png
11
4.VIRTUAL CHARACTER DRIVING PROCESS
The process of driving virtual characters involves several key steps, including skeleton binding,
bone weight modification, and inverse kinematics adjustments using pose data.
Character Skeleton Binding in Blender
• Import T-pose character model.
• Establish bone parent-child relationships.
• Modify bone weights.
• Complete the binding process.
Inverse Kinematics and Clothing Swap System
• Use scripts to retrieve UDP data for the current frame.
• Adjust bone rotations and positions in the LateUpdate() function based on keypoint
data.
Clothing Swap System
• Created using Marvelous Designer.
• UI includes a clothing panel for users to switch outfits seamlessly.
preencoded.png
12
RESULTS
1 Kalman Filtering
Significantly smooths keypoint data.
Without Filtering: Fluctuations noted in tracking.
With Filtering: Achieves a smoother trajectory.
2 Video stream capture methods
From File Path:
Achieved average frame rate: ≥ 120 FPS.
Using Monocular Camera:
Achieved average frame rate: ≥ 130 FPS.
3 Subjective User Evaluation
Precision: 89-93
Real-Time Performance: 94-97
Motion Smoothness: 95-96
preencoded.png
13
RESULTS(CONTD)
preencoded.png
14
FUTURE ENHANCEMENTS
Improved Pose Estimation
Leveraging emerging technologies like depth sensors and multi-view
camera systems can further enhance the accuracy and robustness of
human pose estimation.
Realistic Clothing Simulation
Advancements in physics-based clothing simulation and integration
with virtual characters can elevate the realism of the virtual clothing
displays.
Augmented Reality Integration
Integrating the virtual clothing system with augmented reality (AR)
devices can enable seamless try-on experiences in the real world.
preencoded.png
15
CONCLUSION
Transformative Shopping
The proposed real-time clothing virtual display system paves the way for a transformative
future of online and in-store shopping experiences.
Immersive AR Interactions
The integration of advanced pose estimation and virtual clothing simulations with
augmented reality technologies opens up new possibilities for seamless and engaging
virtual try-on experiences.
Enhanced Customer Satisfaction
By reducing decision fatigue and providing a more personalized and intuitive shopping
experience, this system has the potential to significantly improve customer satisfaction and
loyalty.
preencoded.png
16
REFERENCES
[1] KRIZHEVSKY A, SUTSKEVER I, HINTON G E. Image net classification with deep convolutional neural
networks. Communications of the ACM. 2012; 60(6): 84-90.
[2] Andriluka M, Pishchulin L, Gehler P, et al. 2D human pose estimation: New benchmark and state of
the art analysis. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition;
2014. p.3686-3693. doi: 10.1109/CVPR.2014.471.
[3] Lin TY, Maire M, Belongie S, et al. Microsoft coco: Common objects in context. Proceedings, Part V
13. Computer Vision–ECCV 2014: 13th European Conference; 2014 September 6-12; Zurich,
Switzerland: Springer International Publishing; c2014. P. 740-755.
[4]Ionescu C, Papava D, Olaru V, et al. Human3.6m: Large scale datasets and predictive methods for 3d
human sensing in natural environments. IEEE Transactions on Pattern Analysis and Machine
Intelligence. 2013; 36(7): 1325-1339.
[5] K. Yinghui, Q. Yinfeng, and Z. Ke, "Deep learning based two-dimension human pose estimation: a
critical analysis," Journal of Image and Graphics, vol. 28(07), 2022, doi: 10. 11834/jig. 220436.
preencoded.png
17
THANK YOU
preencoded.png
18
