Because of the snow storm last week, we started off this week with our postponed Quarters session, where we got very useful feedback from faculty about our current direction. We are generally very happy with our theming, but are concerned that we may be a little over-scoped. Hopefully, if we are, it is only by a little, and we believe we can cut back later in the semester if we need to. We did get some very useful previous ETC projects to look into, including BigMR (a project which explored multiplayer MR in a large shared space) and ImmerX (a project which focused on blending the physical and virtual worlds in interesting ways), which will both be very useful inspirations going forward. Another very consistent piece of feedback we got was that we need to define “magical” in the context of our experience – it is a very central pillar of our project, but is a very vague concept that can have different meanings in different situations. This is true, and is something that we will put more intentional thought into going forward.
We also started to advance our tech in three areas: smoothing the ArUco marker data, investigating photogrammetry, and finishing up the Door Demo. We found that the raw data from the ArUco detector was not perfect, and was often a little jumpy. To combat this, we looked into a few different filters for the data, with the goal of making the pose estimations smoother from frame to frame. These filters include a Kalman filter, an interpolation filter, and a moving average filter. While we have these largely implemented, more testing is needed to determine which is optimal for our purposes.
Since we want to mess with the physical world, we also started to look into photogrammetry. Photogrammetry is a method to create a model of a 3D space – by analyzing many 2D images of the space, similar features can be detected in each image and the images stitched together to form one single 3D model. If we are able to get this realistic enough, we believe we could do interesting things with our room, such as flipping it upside down when the guest walks in.
Finally, we finished up our door demo. This involved getting the rotation data from the Raspberry Pi onto the Quest wirelessly, hooking up the sensor to the door itself, and using that data to occlude a virtual environment, each of which had its own challenges. We had to set up a custom network server for the Raspberry Pi to publish data to. This server had to work around restrictions placed on CMU’s network, but we were able to work around that with the help of the ETC tech team (thank you!). Next, we had to find a way to capture the door rotation. We set up a rudimentary link between the door hinge and the rotation sensor using materials from the paint room (pictured below). While this was far from perfect, it would serve as a proof-of-concept. This step also involved creating the calibration curve, to translate the values we got from the rotation sensor into the actual number of degrees the door was open. Finally, we integrated these together in a Unity scene, modelling the door as a rectangle with a special shader on it which would show the real world instead of the virtual. We used more of this shader outside of the room to show the physical hallway, then placed a virtual environment inside the room. With the rotation sensor providing the angle the door was at, we were able to rotate the rectangle to approximately match the angle the physical door was at, occluding the virtual environment the way the actual door would occlude the physical environment behind it.
This demo definitely could use some refinement – there is significant lag and accuracy issues with getting the rotation data, which ruins the illusion. We also did not integrate the fiducial markers into it, as we are still working on them, meaning alignment was an issue. We learned a lot though – we cannot do anything from sensors which require a high level of accuracy, as even relatively small errors in positioning can break the illusion we want to go for. But, we do have the basic structure set up to use more external sensors in the future and to do more with occlusion in special ways.