Repository navigation
What are you using ROS# for? #20
Description
Activity
Hi all,
I'm a master's student at UFMG, in Brazil. My master's thesis project is about robot swarm teleoperation, and since we had a Oculus Rift laying around we thought it was a good idea to use it. However, as you may know, Oculus Rift does not have a fully functional Linux driver and they don't have plans to officially support it, so I made numerous attempts on somehow doing something in ROS and being able to visualize it on Oculus Rift. Maybe I lost 6 months trying different stuff.
In one of my attempts, I started using Unity, and I was looking for some C# libraries for ROS and I stumbled upon ROS#. And it solved all my problems!
You guys had a URDF importer (something I haven't even thought I needed but it was extremely useful). And had also implemented rosbridge with some message types that I needed. In a few HOURS I could implement what I was trying to accomplish for months. 😄Since then, I'm using ROS# to transfer data between Unity and ROS. From Unity I need the Oculus Poses and buttons states. And ROS sends back the positions of my dear little robots and some other visualization stuff.
Reacted by Siyou PeiHi,
I like to test control and visual processing algorithms on robotic platforms. While we can certainly test them on real platforms, it is clearly easier to test them on simulated robots.
To achieve this purpose, I used to work with V-REP interfaced with ROS. However, it turns that a particular project, I needed to get some realistic shading, light reflections, etc... Why my limited knowledge, I think that this is not easily achieved with VREP. I then turned to game engines like Unreal Engine and more recently Unity.
Unfortunately, I initially forgot quickly about Unity, wrongly thinking it is was not possible to use without a license. I therefore invested some time trying to achieve my purpose with Unreal Engine, especially trying to use Microsoft AirSim (yes I needed a drone as well) ; But, the Unreal Engine really requires a beast, not a computer :) And I even do not know how easily it would have been to interface with the ROS world, but definitely the time it takes to compile the engine makes the progresses at a very slow pace. More recently, colleagues suggested to use Unity and it turns out that 1) it is very quick to set up and to run even on a moderately decent hardware and 2) combined with ROS-sharp it seems to perfectly fit my needs.
So, for visually controlled robots, I can benefit from the Unity 3D capabilities for the rendering, and still develop/use algorithms on the ROS side; And this is also more funny to play with nice looking simulations :)
I am a student at the KIT Karlsruhe trying to control a Kuka Robot with the Hololens. Right now we try to use the ros-sharp library but UWP is resilient. So far we only used ros-sharp urdf importer to get our robot into unity because it works better than the old urdf importer. After the import we have to delete ros-sharp because we can't compile the project.
The project goal is to control a robot virtually and only if the robot is able to do a movement without colliding in the virtual world the robot should move in the real world.Hi @Jasonthefirst , thanks for sharing your project with us! We also had a small project with the Hololens this year. Unfortunately without ROS#, though 😉
Let's discuss platform requirements in issue #21. Or, alternatively, please open a new issue if more appropriate.
I find it fascinating for what different types af applications, robots and VR/AR equipment you guys use ROS#. Looking forward to hear more stories!
We are very proud and happy that parts of ROS# already found their way into Process Simulate, a Siemens Simulation Software for production lines.
My colleagues just sent me the link to this youtube video illustrating the import of a URDF model.
Hi @MartinBischoff , here is a follow up with a particular example of Unity3D + RosSharp where we want to simulate underground environments explored with drones.
You can have a look to it at : https://youtu.be/XajgNfNJ1VI
Not the full code is yet provided. Only the code of the drone is available for the moment at : https://github.com/jeremyfix/ros-and-unity/tree/master/DronePrefab
Very impressive @jeremyfix ! Thank you for sharing this info here.
I am happy to see what you can do with ROS#. Are you using this setup to test any (autonomous) functions of the drone? Is the similarity of real and virtual camera data of importance for you here?All: Please do not hesitate and tell us about your applications here. It needn't be as elaborated as the applications above. We are also happy to read about any planned or ongoing tryouts...
@MartinBischoff I indeed use this setup for testing autonomous navigation in underground . This navigation is based on optic flow. For easily testing this ability , we needed to get a simulated environment with 3D shapes, textures, lightning so be as fair as possible with what we may encounter in real underground environements.
Hi all,
I'm an undergraduate from Brown University in Providence, and my lab currently uses ROS# for a variety or projects including VR and AR visualization and teleoperation of a Rethink Robotics Baxter, and a Kinova Robotics Movo.
@MartinBischoff, I'm currently attempting to implement something similar to your turtlebot teleoperation project, but on a different robot. Is the Unity Project for this online somewhere, or is there a tutorial on how to integrate a depth image with the rgb camera? That'd be very helpful!
Our lab will post links to videos and images of the project soon! Excited to share more with all of you.
Hi @NishanthJKumar ! Thanks for sharing this info about your project here!
I am looking forward to see the videos and images!We did not publish our code for the turtlebot application open source. It would be too much effort to to make the code readable, document, fix bugs and administrated it properly. I hope you understand.
For the 3D point cloud we proceeded as follows:
- We used opencv library to uncompress the CompressedImage data.
- We used a script like this to get the points in 3D from the two 2D images.
- We then put this data into a mesh with a simple shader that illustrates point data only.
- By only updating parts of the mesh (e.g. 20%) in each frame you can improve the performance of your application.
Reacted by David Whitney, kosen, Chao Li, stevensu1838 and Xianhao Carton LiuHello everyone,
I am a Masters student in Computer Science from University of Tartu Estonia. I am currently doing an Internship in an Estonian Robotics Organization. My task here is to integrate ROS with Unity to display statistics and other operations on a Hololens device.
So far it was going good with ROS# Library but I had to abandon it. The problem arrises with sockets. The current version of ROS# doesn't work with UWP. Infact System.net.sockets is not working with UWP. After digging deep I found out that UDW supports Windows.Networking for sockets but Unity cannot recognize this. So right now trying to find a workaround to get pass this issue.
Btw I followed your thread for ROS# upgrade in which you use IProtocol to pass socket type on connection but still the issue persisted.
Hello,
I'm currently working on my project where I have to implement a way to teleoperate a Baxter Robot machine using a HTC Vive on a Single Linux Ubuntu machine. Honestly a really daunting task as it's my first year doing coding but I was mainly just using java so C# is thankfully not too far-off. Anyways, I have got Unity working on my machine & learned basics of ROS, now I want to start communicating between the two.
This seems to be exactly the tool I need, so thank you so much for providing this! But I just want to make sure this is actually possible to do so on a single Linux machine as the documentation only mentions a Windows build.
I've seen posts of both people successfully and unsuccessfully using ROS# on a Linux machine... I can't get an opportunity to test this out until Monday but just want to know what other people's experiences with ROS# on a Linux machine is like. Using a Linux Ubuntu 16.04 btw. Thanks for any input! :)
Hi @MartinBischoff ,
First, thanks for making this awesome project available! I'm trying to achieve something very similar to what you've done with the TurtleBot and Vive, but all in Linux. I already have the correct setup for integrating Vive, Unity and ROS using ROS#, but only for the standard message types provided on the unity package. Could you, or someone else who also achieved this, elaborate a bit more on how you handled point cloud/RGBD data coming from ROS and used meshes to render it? I'm a beginner in Unity and couldn't quite get the idea.
Hi everyone,
I am a researcher based in Taiwan National Chengchi University, we are currently developing AI for drones, and my task is to build a Ground Control Station to monitor the drones and supply other members with visual representations of data.
I am actually a game designer and quite familiar with Unity3D but lack the domain knowledge of drones and ROS, had been dealing with QGroundControl for the past few weeks but progress is slow, so I was searching for an alternate solution until I found ROS#.
I am using Unity3D on Ubuntu 16.04 and trying to communicate with my Intel Aero RTF Drone with ROS onboard; Any suggestions are welcomes :D
22 remaining items
Thanks @saltyseabastard for dropping a line here! Happy to read this SW is of use for you.
Hi 👋
I was told that I am allowed to do a little advertisements for my fork. I currently maintain the UWP support fork of this repository over at https://github.com/EricVoll/ros-sharp/
This Ros# version can be deployed to UWP apps and with that to HoloLens 1&2
I recently updated the fork from the previous maintainer and the UWP fork is now almost up to date with the original Ros# and should also contain all features. It is of course fully compatible with MRTK (Version 2.3 & 2.4 tested)Thanks again for @MartinBischoff for your work.
Reacted by Martin Bischoff, ANG0-0GNA and Léon ZeitlerReacted by Thomas GroechelRecently, we initiated a new open source project on github/siemens that we named Evaluation Framework.
It consists of three self-contained tools which facilitate exploring many design variants and finding the best compromise solution.Evaluation Framework can also be used in combination with ROS# as demonstrated here.
Thanks go to Michael Dyck for his commitment in this project.
Check out this article for new ideas on what you can do with ROS#.
Hello! I'm here to thank you for the contribution to ROS#
and shared my project which is "ARCore and ROS integration with ROS#"For someone who wondered if this tool is suitable for android devices
Please check out the video.
https://www.youtube.com/watch?v=28zOEsvyGYMAndroid Set up
https://github.com/petpetpeter/ARrosCoreReacted by Martin BischoffOnly wanted to thank you for this amazing repository and share that we have been working with it to train reinforcement learning neural network which with we allowed a platform to get close to a target position.
And as doctoral student, I am also working with the Hololens 2 adapted fork in order to program robotic arms in an easier way (still work-in-progress).
Reacted by Martin BischoffThank you for your feedback and for sharing this info @Rive4 ! A beatiful scene image and a nice AI application.
Feel free to share further info here, e.g. when you finished your thesis or when you have published a demo video.Hi, @MartinBischoff ，I am impressed by your project, and lately, I took on a project about VR and the turtlebot2 robot. I wanna know how to realize VR to control the robot in real-time?
Hi, I'm a french student and I'm currently working on a sonar. For my project I'm suppose to export data from ROS to Unity, so I choose to use ROS#. However, there is no script to collect this kind of data.
I'm not really familiar with C#, so I decided to "copy" an existing script and understand every part of it.
I created a new MessageTypes for my sonar in order to get and set data, but when I declare the constructor I get an error (CS0824). here you can find a screenshot of my script to create the new MessageTypes:
Could someone explain me why that doesn't work ?
Thanks in advance@simongle99 this is not the right place to ask this question. I'd rather open a new Issue for this :)
But quick answer: remove theexternkeyword and actually implement the constructor.- Sorry I didn't know where to post it. Thank you for your answer…________________________________ De : Eric Voll ***@***.***> Envoyé : mercredi 11 mai 2022 09:05 À : siemens/ros-sharp ***@***.***> Cc : GUILLE Simon ***@***.***>; Mention ***@***.***> Objet : [Externe]Re: [siemens/ros-sharp] What are you using ROS# for? (#20) AVERTISSEMENT: Cet e-mail provient de l'extérieur de l'organisation. Ne cliquez pas sur les liens et n'ouvrez les pièces jointes que si vous connaissez l'expéditeur et que leur contenu est sûr. @simongle99<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fsimongle99&data=05%7C01%7Csimon.guille%40ensam.eu%7C134bbb21fa8047d93a0d08da331ca242%7Ce034b9b077684b9691b2d8f039816ac1%7C0%7C0%7C637878495318550890%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=C62GRPj%2BnS6pGPUI5N7CAL2Xn7CWBBAqMSqDfYRg%2Fck%3D&reserved=0> this is not the right place to ask this question. I'd rather open a new Issue for this :) But quick answer: remove the extern keyword and actually implement the constructor. — Reply to this email directly, view it on GitHub<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fsiemens%2Fros-sharp%2Fissues%2F20%23issuecomment-1123262062&data=05%7C01%7Csimon.guille%40ensam.eu%7C134bbb21fa8047d93a0d08da331ca242%7Ce034b9b077684b9691b2d8f039816ac1%7C0%7C0%7C637878495318550890%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=8GIlWqarSxD%2Br%2BSjgjLOigdWME8OU%2BzspWCoJ38Swbw%3D&reserved=0>, or unsubscribe<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAYDQ5IJZBD7T52BMWHU7BEDVJNL3RANCNFSM4EFGOPTA&data=05%7C01%7Csimon.guille%40ensam.eu%7C134bbb21fa8047d93a0d08da331ca242%7Ce034b9b077684b9691b2d8f039816ac1%7C0%7C0%7C637878495318550890%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2BJuRCHIRECWigvj7EKVA4vNbqPrXskxn2kxFoFGpcQo%3D&reserved=0>. You are receiving this because you were mentioned.Message ID: ***@***.***>
Hey! It looks like I forgot to mention my project NERVV that I developed in 2020 while at Purdue University's Jun Laboratory. While I don't have any demo videos handy, NERVV lets you easily set up machines in Unity, parse input data to create a virtual twin, and then even output this data to control machines with ROS!
Using a framework like NERVV to set up machines gives some benefits like being able to quickly get input and output data flowing easily, but also lets other cool things be built that don't depend on a specific machine or setup as well! For instance, I was able to implement virtual twins with cool features like (additional) collision detection, continuous interpolation, or with the ability to use inverse kinematics that was able to work on any of the NERVV machines.
Then in the NERVV example project, I was able to use that to build a VR demo app that let you view live machine data, adjust any NERVV machine axes manually, or activate IK on any machine to approximate your VR controller's location and orientation! All of this was made possible by ROS# which was really easy to use, so thank you so much!
Unfortunately I don't have any cool videos of using the VR controllers on hand, but here you can see some of the virtual twinned machines in this photo, and these were being streamed live, I promise 😅
Reacted by Martin BischoffHi all, I would like to share my graduation project. Using ROS# (with ROS2), I designed/simulated/prototyped a multi-purpose remote-controlled underwater vehicle. Unity Engine is used for the simulation and remote control Android app, and MATLAB (with ROS Toolbox) is used for the MIMO controller.
unity_sim_demo_compressed.mp4
Here is the repo link: https://github.com/memrecakal/Design-of-a-Multi-Purpose-Remote-Controlled-Underwater-Vehicle-Using-ROS-2-Unity-and-MATLAB-/tree/main
Reacted by Martin BischoffHi all, I would like to share my personal project aiming to build a motion planning and control software to control my first 3DOF arm manipulator using ROS#, Ros2, and Godot.
Reacted by Martin Bischoff and Xianhao Carton LiuVery nice @Anas-mo101 ! Thank you for sharing!
We are very curious what you use ROS# for!
Let us use this issue to present our projects, whether finished, ongoing, or first idea.
I start! Here is our first public project:
A Tutlebot 2 is teleoperated via touchpad controllers HTC Vive.
A 3d point cloud of the camera image is illustrated in VR, as well as the mapping data and the moving turtlebot. The wheels of the turtlebot are animated via JointState messages. Users can teleport to different positions in the scene.
Further, users can switch between teleoperation and simulation mode in real time. In the simulation mode, RosBridgeClient disconnects and Users control the Urdf Simulation model by applying a torque at the wheels. It feels quite similar with the VR glasses on. The only difference is that the camera image is missing and the map does not update anymore. 😏