- The purpose of this project is to transform the text to Cue Speech. In other words, is a text to video task
- The demo of the project divided into two parts. The first parts is points location generation, the second part is the points displacement
- There still question to merge hands with face. The future method will be merge a new blender repo to our project
Google-Mediapipe-Main (cite: MediaPipe: A Framework for Building Perception Pipelines https://arxiv.org/pdf/1906.08172.pdf)
- This folder store the essential code to transform a video body movement to corresponding key points location
- The question of the google-mediapipe is that the output will be divided into three part, body, face, and hands
- The output format of those three are not the same
- This folder is transform the points generate by google-mediapipe to a blender animation
- The question of the points_import is that it cannot merge three outputs of mediapipe correctly, sometime need to adjust by hands
BlenderArMocap (cite: https://github.com/cgtinker/BlendArMocap)
- This folder is a plug in develop by another group, the components of this porject will solve our problem
- However, BlenderArMocap is too large for one to completely understand
| Detection | Output 3D location |
|---|---|
| Import to Blender | Render Motion with Predefined Model |
|---|---|
- After do some improvement to the SRIBD project, use it as a project in CSC3185
Improvement2: Whisper (cite this paper: Robust Speech Recognition via Large-Scale Weak Supervision https://cdn.openai.com/papers/whisper.pdf)
- transcrip people's voice to text
- test code in whisper.ipynb
Improvement3: Sentiment Analysis (cite this paper: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding https://arxiv.org/abs/1810.04805)
- Do the compueter vision analysis using the feature extracted from the video sequence
- Do the natural language processing using the BERT model analize the text extract from the video sequence
| Expression Analysis through Video | Expression Analysis through Audio |
|---|---|
- improving the connection of hands and faces by reading the file of blenderArMocap
- adding the generation model to the google-mediapipe