A simple and easy-to-use library built in TypeScript for real-time lipsyncing with JS and web audio API. Ideal for animating 2D/3D characters in web apps, chatbots, and games.

The examples shows how to use the library with a 3D model in Three.js with React Three Fiber.

Live demo - Video tutorial

npm install wawa-lipsync

or

yarn add wawa-lipsync

First, import the library and create a Lipsync instance:

import { Lipsync } from "wawa-lipsync";

export const lipsyncManager = new Lipsync();

Then, you need to connect the HTML <audio> element to the lipsyncManager:

const audioElement = new Audio("path/to/your/audio/file.mp3");

lipsyncManager.connectAudio(audioElement);

The AnalyserNode used by the library requires the audio element to have a valid source before the connection.

This will work:

const audioElement = new Audio();
audioElement.src = "path/to/your/audio/file.mp3";
lipsyncManager.connectAudio(audioElement);

But this will not:

const audioElement = new Audio();
lipsyncManager.connectAudio(audioElement);

After connecting the audio element, you can start the lipsyncing process by calling the processAudio method at the desired interval:

const analyzeAudio = () => {
  requestAnimationFrame(analyzeAudio);
  lipsyncManager.processAudio();
  const viseme = lipsyncManager.viseme;
  // You can now use the viseme for your animation logic
  console.log(viseme);
};

analyzeAudio();

You can find a complete example of how to use the library with Three.js and React Three Fiber in the examples/lipsync-demo directory of the repository.

If you need help, join the Discord server or open an issue on the GitHub repository.

Contributions are welcome! If you have any ideas, suggestions, or improvements, feel free to open an issue or submit a pull request on the GitHub repository.

This project is licensed under the MIT License. See the LICENSE for more details.