Design Production
From Concept to Experience
This week, Beyond Frame moved from a broad concept toward a much more defined visitor experience.
After discussions with our client, instructors, and team, we began narrowing down not only what the experience should look like, but also how visitors would enter it, move through it, interact with the artwork, and eventually leave with something physical from the experience.
Our current experience flow is becoming much clearer:
Visitors first encounter the original artwork → walk through a physical frame → see themselves integrated into a digital version of the painting → explore the space through natural movement → trigger subtle reactions within the artwork → receive one personalized printed image from their experience.
https://drive.google.com/file/d/1XPtjlmYlUn2B3hbY8tk1cl_kfDBdQXNl/view?usp=sharing
Refining the Interaction
One of the biggest design questions this week was how interactive the experience should actually be.
Early prototypes explored more explicit gesture-based interactions, such as waving, moving arms, or performing different poses. However, through conversations with our instructors and client, we realized that this could easily turn the experience into a gesture game rather than an experience about entering and appreciating an artwork.
Instead, we are now focusing on natural movement.
Walking closer to something, moving behind a tree, leaning around an object, or simply changing position in the space could create small reactions inside the painting.
The client also gave us an important design principle:
Rather than adding new visual effects to the artwork, we should animate what already exists within it.
For example, leaves could move slightly when someone walks nearby, branches could respond subtly to motion, or the visitor could disappear partially behind foreground objects.
The goal is not to make the painting visually louder, but to make it feel quietly alive.
Creating Depth Inside the Painting
One of the strongest directions to emerge this week is the use of depth and layering.
Instead of treating the painting as a flat background with the visitor placed on top of it, we want to separate the artwork into foreground, middle-ground, and background elements.
This would allow visitors to appear inside the composition.
They could walk behind trees, appear between different visual layers, or become partially hidden by objects already present in the painting.
This simple spatial relationship may be one of the most important ways we can create the feeling of actually entering the world of the artwork.
Building the Technical Pipeline
At the same time, we began developing the technical systems needed to make this experience possible.
On the visitor side, we are currently testing body detection, body segmentation, and pose tracking. These systems allow us to locate a visitor in the physical space and separate their body from the camera background.
We can then apply a painting-inspired style to the visitor before placing them inside the digital artwork.
Our current pipeline is roughly:
Camera → Body Detection / Segmentation → Style Transfer → Digital Painting
We already have basic detection and segmentation working, which gives us a foundation for testing how a visitor can move through the digital environment.
We have also been testing different levels of style transfer.
One challenge is finding the right balance between making the visitor visually belong inside the artwork and keeping them recognizable.
If the stylization is too light, the person can look like a photograph pasted onto a painting. If it is too strong, visitors may no longer recognize themselves.
For the next stage, we plan to compare several levels of stylization and find a middle ground between these two extremes.
Separating the Artwork
The painting itself also needs to become something we can manipulate.
We are currently experimenting with a workflow that uses AI-assisted segmentation to separate elements such as trees, branches, ground, water, and other foreground objects into individual layers.
The current process is:
Original Painting → AI Segmentation → Extracted Elements → Artist Cleanup → Background Reconstruction → Unity
This process is still experimental.
AI can help us quickly identify and extract parts of a painting, but the results are not always complete. Objects may need to be manually reconstructed, and removing them can leave missing areas in the background that need to be repaired.
Rather than expecting the entire process to be automatic, we are currently treating it as a hybrid workflow in which AI provides a starting point and artists finish the image manually.
This workflow is also influencing which paintings may work best for the final experience.
Paintings with clear depth, distinct foreground objects, open areas for visitors, and strong lighting conditions are currently much easier to imagine as interactive spaces.
Designing a Spatial Experience
Another important realization this week is that Beyond Frame should not be designed only as an interaction on a screen.
The physical space around the screen is also part of the experience.
Visitors need to understand where to walk, when the experience begins, where they can stand, and when it ends—ideally without reading a large instruction screen.
This means we are beginning to think about environmental guidance through elements such as lighting, sound, floor placement, the physical frame, and spatial layout.
We also explored several possible testing spaces around ETC. A larger, reconfigurable room would allow us to test visitor movement much more realistically than a traditional desktop prototype.
For now, we are intentionally keeping the physical installation simple.
A more elaborate prototype involving touchable physical trees and branches was discussed, but the client felt that it could distract from the painting and introduce unnecessary maintenance and safety issues.
Our priority is to make the screen-based spatial experience compelling first, then consider whether smaller sensory elements such as sound or lighting can strengthen it later.
One Moment to Take Away
The final printed image is also becoming a much more important part of the experience.
Rather than taking many photos and asking visitors to select their favorite, the current direction is to capture one moment that naturally happens during the experience.
The goal is for the image to feel less like posing for a camera and more like a record of something that happened while the visitor was inside the painting.
The client described the intended feeling as:
“This was my moment in the painting.”
We are currently considering formats such as a postcard or photo print. A postcard is particularly interesting because it can function both as a personal keepsake and as something visitors can share with others after leaving the museum.
Moving Toward a Playable Prototype
By the end of Week 2, the project has started moving away from abstract discussion and toward a set of concrete questions that we can answer through prototyping.
The next major goal is not to build every interaction immediately.
Instead, we want to create the simplest possible spatial prototype that can test the core experience:
A visitor enters the space → the system detects them → they appear inside the painting → they move → their representation moves through the artwork.
From there, we can begin testing depth, natural interaction, stylization, sound, and the final image capture.
As we prepare for Quarters, we also want to spend less time explaining how we arrived at the idea and more time showing what we have already built, what the experience currently feels like, and what still needs to be tested.
The most important question moving forward is now very simple:
What does it actually feel like to move through a painting and see yourself become part of it?