MIT XR
Hackathon 2020
Our team created uTooth, a bridge to connect any bluetooth device to your XR experiences in Unity or Unreal.
My role
User Experience and Interaction. Unity 3D demo dev.
How was the
process like?
Hackathon hectic rhythm, constant prototyping in parallel with development.
Everything was done in the course of 4 days of constant work, learn, and iteration.
Who is our user?
Makers, Designers, Hackers, Developers, Curious, Entrepreneurs.
Anyone interested in integrating Bluetooth devices into XR experiences.
Outcome
Our team was one of the top 5 best VR projects at MIT XR Hackathon 2020.
Our project was showcased in the Wild West category due to the all the non conventional devices.
Saying MIT XR Hackathon is the biggest and most diverse developer event in XR is to fall short. This was my second year as a participant, and I can say I got hooked since the first minute I stepped on the event. I always attend MIT XR Hackathon not knowing who I'm going to work with or what I'm going to build and that is absolutely part of the beauty of it. My team this year was formed by Nasa Engineers Tylor Waddell, Alireza Bahremand and designer polymath Mónica Ceisel.
Why?
We wanted to solve a problem that many XR creators stumble upon: currently there is no standardized open framework for Bluetooth communication for most XR engines for external Bluetooth devices and peripherals.
Our team created a framework in C++ and C# for Bluetooth communication between Unity3D and external devices. Unity 3D is one the most used engines for making Mixed Reality Experiences.
Inspiration
As hackers, as creators, we face the limits of technology often, although there are certain boundaries that we are ready to overcome. In our case, we found a problem we want to solve: we want to connect Reality one device at a time.
We’ve created a framework in C++ and C# for Bluetooth communication between Unity and external devices and tested it with Arduino sensors. This is a tool that didn’t exist and allows developers/creators to shorten the implementation time for Bluetooth dramatically.
Accelerating The Pipeline
The current pipeline for XR creators to integrate any Bluetooth devices into Unity 3D or Unreal engines can take an average of 3 days. Loading libraries, configuring hardware, formatting data, setting up communication and setting up devices will take most of the time of any XR creator.
Our framework comes in an open-source and simple frustration-free package, saving XR creators time, money and resources to incorporate any device
The Demo
To test our framework I proposed creating a unique demo, where we’ve connected a living plant with different sensors (moisture, humidity, light…) These sensors connect in real-time with a data visualization experience in VR.
In the demo, the user is able to see and feel how the plant is feeling thanks to a series of particle systems that reflect in real-time the state of our plant.
You may think that a plant can’t make much difference, but our framework will allow developers, creators and makers to connect all kinds of systems from entire aquaponics vertical farms, new custom made controllers for diversely abled users or new tools for surgeons, among many other uses.
How We Built it
We worked in parallel development. Taylor and Alireza developed, tested, & debugged several .NET native Bluetooth libraries to integrate microcontroller communication with Unity for XR projects. While Pilar and Mónica worked on the demo scene, search and tweak assets, and wrote the scripts that will interact with the demo plant. Finally, we merged our two separate scenes together with Unity BT scene and spent time debugging functionality.
Challenges We Ran Into
The main issue we ran into was the System.IO.Ports namespace for C#/Unity. For this class it was able to maintain an open BlueTooth connection, it would briefly connect and then drop. We believe this comes from a complex threading issue of how the Serial class maintains its connection. To make sure our project still succeeded we used a USB communication as windows binds these to COM ports the same way it does with BlueTooth. This allowed us to make a framework that works the exact same way. The other issue is since we had a lot of personal hardware there were many different ways in which we had to interface ranging from the sensor data to communication protocols.
We learned a lot about the lower levels of C# programming, and how to use lower-level functions.
What Is Next
What is next is to either wait till System.IO.Ports is updated our create our own serial communication libraries which may be easier. The data transfer was also much slower than it could be, sending smarter packers and moving around data in a more efficient way would allow for much high fidelity sensors such as videos and audio.
_JPG.jpg)