ETC Playtest Day

On Saturday CMU ETC has an important event playtest day there will be testers out of ETC coming to the building and playing the projects so for week 8 we were mainly preparing for the playtest event. 

Playtest Structure

In this playtest session, our goal is to gain insights on:

• The effectiveness of the educational game – they understand the knowledge so they get the answer right
• The validity of stealth assessment by data tracking
• The playability of the demos

The participants should be divided into two groups:

• 10-18 year olds (ideally middle school students)
• Adults (mainly for playability test)

This an example of our planned schedule. We will try to get as many as possible results from both Physics and History demos to get an overview of the assessment.

The chosen playtest method include:

Observe

• We will try to find the motive, issue, interest, etc. by collecting their verbals, gestures or reactions.
• We will try to not interrupt them by congratulating, explaining, helping them or apologizing.
• We won’t do a think-aloud at this point because it will interrupt their operation flow.

Interview

• We will ask two main open-ended questions to help us understand the players better and approach our playtest goal.

Survey

• We will ask the players to fill out our surveys designed for physics and history subjects.

We plan to have at least two people in one team to curate one playtest at a time. 

One records and observes the interaction of players, the other takes notes in the way. 

And in the interview stage, one should ask notable questions and the other should take notes.

Data Tracking:

As we mentioned earlier in previous posts we will use the data store inside Roblox for data tracking as the major part of the stealth assessment. The data received will be sent to discord as bot so we can easily gather them and convert them to an excel sheet or JSON file. We will combine the result with the three methods we used during playtest to build a complete analysis of educational assessment.

Demo for playtesting

For the physics prototype we currently have three levels. Similar to the 1/2s verision, the player will discover the friction and push the ball off the ramp and make it touch the button to open the door to the next level.

The history prototype compares to week 7 has a lot of progress. We now have three levels teaching the relationship between different historical events. Players can click ‘E’ to see the description and use ‘R’ to rotate the nodes and point the laser connecting to the right nodes.

Meeting with Jessica Hammer

On Thursday we met with Jessica Hammer, professor at the HCI Institute at CMU. Jessica provides us with a lot of insights on our current two prototypes. We were being asked a lot about what if students randomly try different settings in the levels but still pass it. How to make sure they really learn? Jessica gave us great advice that maybe we can implement a coin system in that the player needs to pick up a coin before changing the friction so the players will stop and think before trying things without looking at the content.

Another suggestion is that maybe when the player finishes the level we can have a character called a ‘teachable agent’ that asks the player why they did that and let the player teach the NPC so we can observe if they really understand the topic.

For the history, Jessica mentioned that instead of having laser maybe we can let the player put digital puzzle pieces so there are more physical actions. And we can also add audio so if they don’t read the text they can still passively hear the contents.

Next Week

For next week we will start analyzing the results & data gathered from playtest day. We will keep working on polishing the demo and assessment process. See you next week!