Category: Blog

Because the project will include multi-room experiences. We believe crossing doors is one of the necessary experiences. To make the door opening action smooth and believable, we are going to combine the features of OpenCV, ARUCO Tags, and Phidgets, and discover how to make these different technologies work together. This research will provide strong support in the future when we install the experiences. Thus, the Door Occlusion Demo is not only a utility research, but also the first step for the team to get through our pipeline.

OpenCV is the go-to computer vision library. Our current plan is to use it for ARUCO Tags, which are physical markers placed around our environment to help us recalibrate position and orientation, combating drift. We have managed to get position and orientation information from the tags, and are now working on positioning an environment based on that information.
For the door Occlusion demo, we are using special shaders that tell the Quest whether to render real-world data or virtual-world data. This way, we can model the position of the hallway and door in our Unity environment, apply this shader, and the Quest will show the user the virtual world only where we want it to

Since we are building an experience in a space that we can control, using external sensors and trackers is something we are exploring. We are using Phidgets to build a prototype with a rotation sensor to track the angle of a door hinge. With this, we can align a virtual door with the physical door, allowing for virtual objects to be occluded by a moving real-world object. The challenge with using Phidgets and the Meta Quest is the wired connectivity. To work around this, we have connected the Phidgets to a Raspberry Pi and are using a network server.

This week, we continued to define what we wanted this project to do and feel like in preparation for Quarters next week. We settled on a theme of liminal spaces – spaces where reality seems slightly off, slightly shifted. These are often large, empty, or abandoned spaces, often dimly lit, and tend to have a very slight horror element. We felt this worked well with the project goals. Because liminal spaces are very open, it works very well with the multi-room project goal. Additionally, there is a sense that weird things happen in liminal spaces, which means we can integrate the magical moments we hope to go for. Finally, liminal means “occupying the position at, or on both sides of, a boundary or threshold.” This is how these spaces get their name – they seem to occupy the space between the real world and a slightly different reality. But, it works really well for our project as well, as mixed reality is all about existing in both the virtual and digital worlds, which will be a major pillar of our project. For these reasons, we decided that our team name would be Liminal.

We also continued to set up this week. We got our perforce folder set up so we can share project files. We set up Meta Quests for the whole team. We worked on an April Tag detector in Unity which works with a webcam to help with quick iteration and debugging. And, we started work on our half-sheet and poster design, which we decided would be centered around a liminal indoor pool, with colorful features in the pool representing our six team members. We presented this rough draft in a design meeting with Charles, and got some good feedback about branding and design.

Finally, we put together our Quarters presentation, clarifying our initial ideas for when faculty will visit our project room and provide early questions and feedback for us to think about as we went forward. We wanted to show our theming and mood board, along with the technology we believe will help us on our way. We hope to get good feedback about feasibility/scope and maybe some interesting directions to take the project within this theme!

This week, we had discussions with our instructors regarding the project’s overall requirements and expected goals. Jesse Schell and Dave Culyba introduced us to the research direction of the project, the scale positioning of the entire floor, and their expectations for us, which provided a clear understanding of the project’s focus. In terms of collaboration within the team, we have set up the basic team information, confirmed the weekly schedules like core hours and weekly meeting time, and clarified the division of responsibilities among each member. We believe that using VR and MR technology to create art on an entire floor is exciting, and this is also one of our main goals. We hope to ultimately produce a “magical” experience similar to LBE or art installations, using MR as the medium, that transcends reality.

To start off our first week of development, our team began by setting up our Meta Quest 3 Headsets and creating a demo. After getting a basic demo set up, letting us know we could utilize our Meta Quest 3 at all, we looked into Meta’s SDKs and samples in order to look at how to access camera data. Once we were confident in that, we eventually integrated a 3rd party library built to track AprilTags in conjunction with the Meta Quest 3’s camera data, where we are now able to gather spatial distance of AprilTags and how far they are from our headset. Next, we are looking at ways to potentially use AprilTags to calibrate objects on the scale of a room, as well as running OpenCV to get access to a wider variety of visual processing.

“What’s an AprilTag?
An AprilTag is a visual fiducial system popular in robotics research, that allows us to track particular patterns for their ID, Location, and Orientation. They’re similar to QR codes!”

In addition, this week we also participated in the “Composition Box” event held by ETC every semester. We have exchanged our thoughts with other project teams, and received many valuable and inspiring pieces of feedback. These suggestions from different perspectives have helped us re-examine the potential improvement directions of the project, and have important reference significance for optimizing the experience design for the following weeks. Mike and John also shared about team collaboration after the event, and we learned a lot from it. These contents will directly guide our work methods in project execution in the future.