Welcome to Ubiq-Genie

Ubiq-Genie is a framework that enables you to build server-assisted collaborative mixed reality applications with Unity using the Ubiq framework. This is particularly useful for building multi-user applications that require server-side processing such as generative models, conversational agents, and real-time transcription. For more information, please refer to the Ubiq-Genie paper.

These instructions will get you a copy of the project up and running to run the samples and to start building your own applications. Ubiq-Genie supports Windows, macOS, and Linux. Ubiq-Genie has a server-client architecture, which means you may need to run the server on a separate machine from the Unity client.

0. Install Node.js (v20 or later) and Python (v3.10 or later).

1. Clone this repository somewhere on your machine (either local or remote).

2. Open a terminal in the Node folder and run npm install to install the dependencies.

3. Install the Python dependencies for the services you plan to use. Each service provider has its own requirements.txt file located in its provider folder (e.g., Node/services/speech_to_text/providers/azure/requirements.txt). Install them with pip install -r <path_to_requirements.txt>. If you are using a virtual environment, activate it before running the command. Missing dependencies will be flagged as warnings when a service starts. Please ensure that you have the correct PyTorch and CUDA versions installed (see the PyTorch website for more information).

Install Unity 6.0 LTS (version currently used is Unity 6000.0.67f1). There are two ways to set up the Unity client:

This is the quickest way to explore the samples.

1. Clone or download this repository if you haven't already, and add the Unity folder to Unity Hub. The Ubiq-Genie package (com.ucl.ubiq-genie) is included as an embedded package and will be loaded automatically.
2. Wait for Unity to finish importing. On first open, Ubiq-Genie will automatically install its dependencies and import Ubiq's Demo (XRI) sample, which adds the XR Interaction Toolkit. This may trigger several editor restarts.
3. In the Project window, navigate to Assets/Ubiq-Genie/Apps/ and open any sample scene.

Use this if you want to integrate Ubiq-Genie into an existing Unity project.

1. In Unity, go to Window → Package Manager → + → Add package from git URL and enter:

   https://github.com/UCL-VR/ubiq-genie.git#upm
   

2. Wait for Unity to finish importing. Ubiq-Genie will automatically install its dependencies. This may trigger several editor restarts.

3. Import any Ubiq-Genie sample from Window → Package Manager → Ubiq-Genie → Samples and open its scene.

For more information on how to use Ubiq-Genie, please refer to the README files in the Node folder.

The Node/apps folder contains a number of samples that demonstrate how to use Ubiq-Genie, which each utilize one or more services defined in the Node/services folder. For more information on how to use these samples, please refer to the README files in the corresponding folders. Currently, the following collaborative sample applications are available:

• Texture Generation: generates a texture based on voice-based input and an optional ray to select target objects
• Multi-user Conversational Agent: a conversational agent that can be interacted with by multiple users
• Transcription: transcribes and audio of each user in the room in separate files

Apps may optionally include version subfolders under Node/apps/<app-name>/, where each version has its own app.ts and config.json. Start with npm start <app-name> to choose interactively, or npm start <app-name> <version> to launch a specific version.

For a demo video of the samples, please refer to the Ubiq-Genie demo video.

For any questions, please use the Discussions tab on GitHub or send a message in the ubiq-genie channel in the Ubiq Discord server. For bug reports, please use the Issues tab on GitHub.