As Halves approaches, Beyond Frame has reached an important transition.

Over the past few weeks, we have focused on making the experience technically possible: placing visitors inside a painting, tracking their movement, creating depth, triggering environmental interactions, capturing an image, and producing a physical print.

This week, our focus shifted toward a different question:

How does a visitor know what to do once they enter the painting?

Our first external playtest revealed that having interactions is not enough. Visitors also need to discover them naturally.

Rather than adding more obvious instructions or game-like UI, we are now exploring how the painting itself—and the physical environment around it—can quietly guide people through the experience.

Taking the Prototype Outside the Project Room

This week, we brought a lightweight version of Beyond Frame to the Hunt Library Playtest Night.

Until now, much of our testing had happened within the team, where everyone already understood how the system was supposed to work.

Testing with new visitors immediately exposed different behaviors.

Many people tended to stay in one place rather than explore the full tracking area. They moved their arms much more often than their entire bodies, moved left and right more naturally than forward and backward, and—unexpectedly—many people tried jumping.

These observations highlighted something important:

Visitors will not necessarily move through the experience simply because the technology allows them to.

The environment needs to give them a reason.

We also asked visitors a broader question:

“If you were inside this painting, what would you want to interact with?”

Their responses included interacting with figures already in the painting, touching or manipulating water, changing brush strokes, adjusting light or color, jumping, grabbing objects, and interacting more physically with the environment.

These ideas are helping us understand what people instinctively expect when they imagine being “inside” an artwork.

Discoverability Became the Main Design Problem

The playtest helped us identify three fundamental questions:

Where do I start?

How do I discover depth and hiding?

How do I know what I can interact with?

Previously, we had been thinking mostly about interaction design from the system side: what the camera can detect, what gestures are reliable, and what elements can respond.

Now we are thinking from the visitor’s perspective.

A tree may technically respond when someone reaches toward it, but if nothing suggests that the tree is interactive, the visitor may never try.

A visitor may be able to walk behind a foreground layer, but if the experience does not encourage forward and backward movement, they may remain in the same position for the entire session.

Using Sound as Guidance

Sound may become one of the most useful tools for guiding visitors without visible instructions.

Instead of simply adding environmental ambience, sound can respond to proximity.

As a visitor approaches a tree, they might begin to hear leaves moving.

As they approach water, the sound of water could become clearer.

A subtle sound could attract someone toward a region of the painting before they understand exactly what will happen there.

This makes sound both an atmospheric layer and an interaction cue.

The client felt that this may be more natural than relying heavily on glowing outlines or other visual indicators.

This week therefore reinforced an idea that has been developing throughout the project:

sound should not simply accompany the painting—it can help visitors understand it.

“Subtle” Does Not Mean “Small”

Another useful discussion this week was about the meaning of subtle interaction.

Earlier in development, we sometimes treated “subtle” as meaning that every response needed to be very small.

Our instructors challenged that assumption.

A larger or more surprising interaction can still belong in Beyond Frame as long as it grows naturally from the artwork itself.

The painting should remain the star.

For example, manipulating an existing branch, changing the light across the landscape, or creating a noticeable response in water could potentially feel more dramatic without turning the experience into a gimmick.

The important distinction is not necessarily:

small interaction vs. large interaction

but rather:

interaction that belongs to the artwork vs. interaction added only for spectacle.

Learning From a Second Painting

Our second painting is also forcing us to reconsider how reusable the interaction system should be.

The first forest painting naturally supports trees, leaves, foreground hiding, and strong depth.

The second painting has a different visual style and composition, including a figure and a bench.

That means simply repeating the same interactions would not make sense.

Instead, different regions or subjects in the painting might behave differently. The existing figure could react or disappear when approached, or depth could be used differently from the forest scene.

This is important for the larger project.

We want Beyond Frame to have a repeatable pipeline, but we do not want every painting to become the same experience with a different background.

The technology can be shared.

The interactions should still emerge from the artwork.

Refining the Art Pipeline

Working with the second painting also exposed limitations in our current image-processing workflow.

Our AI-assisted background reconstruction is functional, but it is not yet clean enough to use without manual correction.

We have seen:

• visible seams between reconstructed areas
• changes in color
• altered proportions
• difficulty separating pointillist or visually complex elements

This reinforced our current approach to AI:

It is useful for accelerating the first stage of preparation, but it does not replace artistic cleanup.

We are also exploring softer or partially transparent edges between layers so that reconstructed elements blend more naturally with the original artwork.

Testing this process on a second painting is particularly important because we do not want our pipeline to work only for one specific image.

Solving Scale and Perspective

Another major technical challenge remains visitor scale.

We need the visitor to appear naturally integrated into the perspective of the painting while still giving them enough physical room to interact.

Those two goals sometimes conflict.

A large physical tracking area gives visitors freedom, but it can make perspective mapping more difficult. A carefully aligned composition looks more convincing, but may restrict movement.

One possible simplification is to give each visitor access to a narrower “slice” of the painting rather than requiring the entire horizontal composition to remain physically reachable.

They could still move forward and backward through depth while interacting with a more controlled area.

This is something we will continue testing rather than solving entirely before Halves.

Rethinking the Physical Print

We also made progress on the physical takeaway.

The Canon Selphy printer works well for prototyping and can produce an image in roughly 24 seconds, which is fast enough for the current visitor flow.

However, its photo paper does not work particularly well as a traditional postcard because the back cannot easily receive our own printed design.

This led us toward a new direction:

photo print + designed envelope or sleeve

The sleeve could contain museum branding, artwork information, or a more graphic interpretation of the frame.

It could potentially be screen printed or risograph printed, making the physical object feel less like a standard photo and more like a small collectible artifact from the experience.

This solution also lets us keep using a simple, reliable photo printer during prototyping without overengineering the printing system.

The Installation Is Coming Together

Meanwhile, the physical installation is becoming increasingly complete.

The monitor is installed, the camera is positioned beneath it, the tracking area has been marked, speakers are in place, the curtain has arrived, and fabrication of the physical frame is progressing.

Lighting remains one of the major pieces still being tested.

The system can detect a segmented body even in relatively low light, but reliable gesture detection still needs to be tested under the final lighting conditions.

This is particularly important because lighting now serves two purposes:

technical visibility and environmental atmosphere.

Finding the right balance will be part of the next installation tests.

Preparing for Halves

With Halves approaching, our priority is deliberately not to expand the project dramatically.

The playtest generated many interesting ideas—from manipulating paint to interacting with figures and water—but they are currently informing our future design rather than becoming an immediate feature list.

Before Halves, we are focusing on:

• improving body and gesture detection
• improving visitor guidance
• integrating sound
• refining scale and perspective
• continuing the second painting
• completing the physical installation
• stabilizing the full experience loop

Our presentation strategy has also changed.

Instead of spending several minutes explaining the concept before showing the work, we plan to introduce the project briefly and then show the experience as early as possible.

A narrated video will demonstrate both an obvious interaction and a more subtle one before we explain the art and technical pipelines behind them.

This mirrors what we learned from the playtest itself.

Beyond Frame becomes much easier to understand when people can see someone enter the painting.