Dev branch PR with updated yolo demo apps created by PhoenixSun17 · Pull Request #10 · UCL-VR/ubiq-genie

PhoenixSun17 · 2026-03-05T17:17:37Z

Appended Yolo Floating label demo, full configured services, apps and unity scene

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Copilot

Pull request overview

Copilot reviewed 99 out of 107 changed files in this pull request and generated 18 comments.

Files not reviewed (1)

Node/package-lock.json: Language not supported

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-03-05T22:38:01Z

+            float adjustedX = pos.x - 960f;
+            float adjustedY = 540f - pos.y;


UpdateElement hard-codes 960f/540f (half of 1920x1080) when converting coordinates. This will place labels incorrectly if the input resolution or the panel size changes. Prefer deriving the offset from targetPanel.rect (and/or the source image dimensions) instead of fixed magic numbers.

Suggested change

float adjustedX = pos.x - 960f;

float adjustedY = 540f - pos.y;

// Derive offsets from the actual targetPanel size instead of hard-coding 1920x1080 halves.

float halfWidth = targetPanel != null ? targetPanel.rect.width * 0.5f : 960f;

float halfHeight = targetPanel != null ? targetPanel.rect.height * 0.5f : 540f;

float adjustedX = pos.x - halfWidth;

float adjustedY = halfHeight - pos.y;

Copilot · 2026-03-05T22:38:02Z

@@ -0,0 +1,21 @@
+import { ServiceController } from '../../components/service';
+import { NetworkScene } from 'ubiq';


NetworkScene is imported from 'ubiq', but the rest of the codebase imports it from 'ubiq-server/ubiq' (e.g. Node/services/base/service.ts). Unless there is an alias/package named ubiq, this will fail at build time. Update the import to match the existing services.

Suggested change

import { NetworkScene } from 'ubiq';

import { NetworkScene } from 'ubiq-server/ubiq';

Copilot · 2026-03-05T22:38:02Z

+# Base Application
+
+This folder contains the code for the base application, which serves as a template for creating new applications using the Ubiq-Genie framework. Please refer to the [README](../README.md) to learn how to create your own applications.
+
+> [!IMPORTANT]
+> Before proceeding, ensure the Ubiq-Genie framework and the necessary dependencies for this sample are correctly installed. For further details, please see the root [README](../../../README.md) file.
+
+## Running the Sample
+
+### Server (Node.js)
+
+We recommend using VS Code to run and modify the server application, with the `Node` folder as the workspace root. To run the server application:
+
+1. Open a terminal and navigate to the `Node/apps/base` directory.
+2. Execute the below command, which will guide you through the configuration process, including setting up the server information. Configuration will only be run the first time you start the application. Ensure that you apply the same server configuration to the Unity client (in `Unity/Assets/ServerConfig.asset`).
+
+    ```bash
+    npm start base
+    ```
+
+If you need to reconfigure the application, you can run `npm start base configure`. You may also manually configure the application by changing the `config.json` which contains the server configuration.
+
+### Client (Unity)
+
+1. Launch Unity and navigate to the `Unity/Assets/Apps/Base` directory. Open the `Base.unity` scene.
+2. Ensure the `Room Client` under the `Network Scene` object has the correct IP address and port for the server. If the server is running on the same machine as the Unity Editor, the IP address should be `localhost`. This should correspond to the configuration on the server side.


This README appears to be copied from the base app: the title is "Base Application" and the run instructions reference Node/apps/base and the Unity Base.unity scene. Update it to accurately describe how to run the DynamicCapture app (correct folder, npm command, and Unity scene).

Copilot · 2026-03-05T22:38:02Z

+m_EditorVersion: 6000.0.61f1
+m_EditorVersionWithRevision: 6000.0.61f1 (74a0adb02c31)


This change downgrades the project Unity editor version from 6000.0.67f1 to 6000.0.61f1, but the repo root README currently states the project uses 6000.0.67f1. Either revert this downgrade or update the documented required Unity version so contributors don’t open the project with the wrong editor and trigger reserialization churn.

Copilot · 2026-03-05T22:38:04Z

@@ -0,0 +1,153 @@
+using System.Collections.Generic;
+using UnityEngine;
+using UnityEngine.UI;


UnityEngine.UI is imported but not used in this file. Please remove unused usings to avoid warnings and keep imports minimal.

Suggested change

using UnityEngine.UI;

Copilot · 2026-03-05T22:38:05Z

+# Base Application
+
+This folder contains the code for the base application, which serves as a template for creating new applications using the Ubiq-Genie framework. Please refer to the [README](../README.md) to learn how to create your own applications.
+
+> [!IMPORTANT]
+> Before proceeding, ensure the Ubiq-Genie framework and the necessary dependencies for this sample are correctly installed. For further details, please see the root [README](../../../README.md) file.
+
+## Running the Sample
+
+### Server (Node.js)
+
+We recommend using VS Code to run and modify the server application, with the `Node` folder as the workspace root. To run the server application:
+
+1. Open a terminal and navigate to the `Node/apps/base` directory.
+2. Execute the below command, which will guide you through the configuration process, including setting up the server information. Configuration will only be run the first time you start the application. Ensure that you apply the same server configuration to the Unity client (in `Unity/Assets/ServerConfig.asset`).
+
+    ```bash
+    npm start base
+    ```
+
+If you need to reconfigure the application, you can run `npm start base configure`. You may also manually configure the application by changing the `config.json` which contains the server configuration.
+
+### Client (Unity)
+
+1. Launch Unity and navigate to the `Unity/Assets/Apps/Base` directory. Open the `Base.unity` scene.
+2. Ensure the `Room Client` under the `Network Scene` object has the correct IP address and port for the server. If the server is running on the same machine as the Unity Editor, the IP address should be `localhost`. This should correspond to the configuration on the server side.


This README appears to be copied from the base app: the title is "Base Application" and the run instructions reference Node/apps/base and the Unity Base.unity scene. Update it to accurately describe how to run the Yolo app (correct folder, npm command, and Unity scene).

Copilot · 2026-03-05T22:38:05Z

+import cv2
+import numpy as np
+import socket
+import json
+import csv
+import time
+
+# --- CONFIGURATION ---
+UDP_IP = "127.0.0.1"
+UDP_PORT = 5065
+UNITY_AREA_WIDTH = 7
+UNITY_AREA_HEIGHT = 7
+LOG_FILENAME = "tracking_log.csv"
+
+# Camera Params (Insta360 - standard or fisheye, used if you uncomment undistort)
+K = np.array([[1.15709813e+03, 0, 9.86153891e+02],
+              [0, 1.17329523e+03, 5.82845627e+02],
+              [0, 0, 1]], dtype=np.float32)
+D = np.array([-0.57465333, 0.64656927, 0.01184943, 0.0153547, -0.39175262], dtype=np.float32)
+
+# Global variables for calibration
+calibration_points = []
+calibration_complete = False
+reference_frame = None  # To store the initial ground truth
+prev_frame = None       # To store the immediate previous frame for motion detection
+
+def mouse_callback(event, x, y, flags, param):
+    global calibration_points, calibration_complete
+    if event == cv2.EVENT_LBUTTONDOWN:
+        if len(calibration_points) < 4:
+            calibration_points.append((x, y))
+            print(f"Point selected: {x}, {y}")
+        if len(calibration_points) == 4 and not calibration_complete:
+            calibration_complete = True
+            print("Area defined! Tracking started.")
+
+def calculate_perspective_matrix(src_points, width, height):
+    src = np.float32(src_points)
+    # Unity Coordinate Mapping (Top-Left Origin)
+    dst = np.float32([[0,0],
+        [width, 0],
+        [width, height],
+        [0, height]])
+    return cv2.getPerspectiveTransform(src, dst)
+
+def map_pixel_to_unity(matrix, pixel_x, pixel_y):
+    point = np.array([[[pixel_x, pixel_y]]], dtype=np.float32)
+    transformed = cv2.perspectiveTransform(point, matrix)
+    return transformed[0][0]
+
+def calculate_unity_scale(matrix, x, y, w, h):
+    """Calculates the width and height of a bounding box in Unity units."""
+    # Transform the top-left and bottom-right points of the bounding box
+    p1 = map_pixel_to_unity(matrix, x, y)
+    p2 = map_pixel_to_unity(matrix, x + w, y + h)
+
+    # Scale is the absolute difference between the transformed coordinates
+    u_w = abs(p2[0] - p1[0])
+    u_h = abs(p2[1] - p1[1])
+    return round(float(u_w), 3), round(float(u_h), 3)
+
+def run_tracking(video_source=0):
+    global reference_frame, prev_frame
+
+    # 1. Setup UDP
+    sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
+


This script imports modules that are unused in the current implementation (socket and time), and creates a UDP socket even though sendto is commented out. Remove unused imports/resources (or re-enable UDP sending) to avoid confusion and to make it clear how output is expected to be delivered to the Node service.

Copilot · 2026-03-05T22:38:05Z

+    # 2. Setup Camera
+    cap = cv2.VideoCapture(video_source, cv2.CAP_DSHOW) 
+    cap.set(cv2.CAP_PROP_FRAME_WIDTH, 1920)
+    cap.set(cv2.CAP_PROP_FRAME_HEIGHT, 1080)


cv2.VideoCapture(video_source, cv2.CAP_DSHOW) forces the DirectShow backend, which is Windows-specific and can prevent capture from working on macOS/Linux (the repo README states cross-platform support). Consider omitting the backend argument or selecting it conditionally based on platform.

Copilot · 2026-03-05T22:38:06Z

+import cv2
+import numpy as np
+import socket
+import json
+import csv
+import time
+from ultralytics import YOLO
+
+# --- CONFIGURATION ---
+UDP_IP = "127.0.0.1"
+UDP_PORT = 5065
+UNITY_AREA_WIDTH = 1920
+UNITY_AREA_HEIGHT = 1080
+LOG_FILENAME = "yolo_tracking_log.csv"
+
+# Global variables for calibration
+calibration_points = [(0,0), (1920,0),(1920,1080),(0,1080)]
+calibration_complete = True
+
+def mouse_callback(event, x, y, flags, param):
+    global calibration_points, calibration_complete
+    if event == cv2.EVENT_LBUTTONDOWN:
+        if len(calibration_points) < 4:
+            calibration_points.append((x, y))
+            print(f"Point selected: {x}, {y}")
+        if len(calibration_points) == 4 and not calibration_complete:
+            calibration_complete = True
+            print("Area defined! YOLO Tracking started.")
+
+def calculate_perspective_matrix(src_points, width = 1920, height = 1080):
+    src = np.float32(src_points)
+    # Unity Coordinate Mapping (Top-Left Origin)
+    dst = np.float32([[0,0],
+        [width, 0],
+        [width, height],
+        [0, height]])
+    return cv2.getPerspectiveTransform(src, dst)
+
+def map_pixel_to_unity(matrix, pixel_x, pixel_y):
+    point = np.array([[[pixel_x, pixel_y]]], dtype=np.float32)
+    transformed = cv2.perspectiveTransform(point, matrix)
+    return transformed[0][0]
+
+def run_tracking(video_source=0):
+    global calibration_complete
+
+    # 1. Setup UDP
+    # sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
+
+    # 2. Setup Camera
+    cap = cv2.VideoCapture(video_source) 
+    cap.set(cv2.CAP_PROP_FRAME_WIDTH, 1920)
+    cap.set(cv2.CAP_PROP_FRAME_HEIGHT, 1080)
+
+    # 3. Load YOLO Model (yolov8n is the nano model, great for real-time webcam)
+    print("Loading YOLO model...")
+    model = YOLO("yolov8n.pt") 
+
+    # 4. Logging Setup


This script imports modules that are unused in the current implementation (socket and time), and the UDP send is commented out. Remove unused imports (or re-enable UDP) to keep the script focused and avoid implying network behavior that doesn't happen.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

PhoenixSun17 and others added 12 commits December 1, 2025 13:15

update my own service

6c052c1

Merge branch 'UCL-VR:main' into main

04fb31d

Merge branch 'UCL-VR:main' into main

895a408

Merge branch 'UCL-VR:main' into main

fbd9233

update dynamic capture

49e630a

update dynamic capture service

e03f593

update new service

28aa6ff

update capture with unity scales

7b696a5

Update Capture.py

97537cd

Merge branch 'main' into dev

cf4c6b8

adding yolo floating label demo

0547504

update unity side for yolo demo

2dea6c6

nsalminen requested review from Copilot and nsalminen March 5, 2026 17:57

Copilot AI reviewed Mar 5, 2026

nsalminen requested a review from Copilot March 5, 2026 22:28

Copilot started reviewing on behalf of nsalminen March 5, 2026 22:28 View session

Copilot AI reviewed Mar 5, 2026

View reviewed changes

PhoenixSun17 and others added 3 commits March 6, 2026 13:48

Update Unity/Assets/Scripts/MessageReceiver.cs

421203d

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

resolve conflict according to PR comments

889f1fa

Merge branch 'main' into dev

e048e12

Navigation Menu

Repository navigation

Dev branch PR with updated yolo demo apps created - #10

Dev branch PR with updated yolo demo apps created#10
PhoenixSun17 wants to merge 15 commits into
UCL-VR:devfrom
PhoenixSun17:dev

PhoenixSun17 commented Mar 5, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI Mar 5, 2026

Uh oh!

Copilot AI Mar 5, 2026

Uh oh!

Copilot AI Mar 5, 2026

Uh oh!

Copilot AI Mar 5, 2026

Uh oh!

Copilot AI Mar 5, 2026

Uh oh!

Copilot AI Mar 5, 2026

Uh oh!

Copilot AI Mar 5, 2026

Uh oh!

Copilot AI Mar 5, 2026

Uh oh!

Copilot AI Mar 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		float adjustedX = pos.x - 960f;
		float adjustedY = 540f - pos.y;

-            float adjustedX = pos.x - 960f;
-            float adjustedY = 540f - pos.y;
+            // Derive offsets from the actual targetPanel size instead of hard-coding 1920x1080 halves.
+            float halfWidth = targetPanel != null ? targetPanel.rect.width * 0.5f : 960f;
+            float halfHeight = targetPanel != null ? targetPanel.rect.height * 0.5f : 540f;
+            float adjustedX = pos.x - halfWidth;
+            float adjustedY = halfHeight - pos.y;

		@@ -0,0 +1,21 @@
		import { ServiceController } from '../../components/service';
		import { NetworkScene } from 'ubiq';

	import { NetworkScene } from 'ubiq';
	import { NetworkScene } from 'ubiq-server/ubiq';

		m_EditorVersion: 6000.0.61f1
		m_EditorVersionWithRevision: 6000.0.61f1 (74a0adb02c31)

Conversation

PhoenixSun17 commented Mar 5, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI Mar 5, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 5, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 5, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 5, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 5, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 5, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 5, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 5, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 5, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants