ChatbotAI Postmortem

ChatbotAI is a student pitch project exploring how generative AI like ChatGPT could be used to make narrative, character driven experiences. To do this we delivered Hysteria in Howlsbend. The description for the game is:

Set in a colonial-era New England town driven to hysteria when a beloved member of the community is afflicted by a witch. You lead the witch hunt by having typed one-on-one conversations with a cast of characters powered by ChatGPT. Can you uncover the mystery and help decide the town’s fate?

Our team members are:

Charlie Agriogianis

Lead Game Designer

Technical Designer

Narrative Designer

Amber Griffith

Lead Narrative Designer

Artist

Dmitry Portnoy

Lead Programmer 

Game Designer

Anna Salieva 

Lead Artist

Narrative Designer

UI Designer

Jingyu Zhuang

Producer

Programmer

What Went Well

We didn’t take the easy route – We did not make this project easy on ourselves from the start. Beyond just the base challenge of taking on new technology, we also took on the challenges of a less explored historical setting, as well as those inherent in interrogation games, a rarely explored genre. We took on the challenge of both being an exploratory and deliverable-focused project. And we took on the additional challenge of focusing on and writing an original story with strong themes and commentary. 

We could have just focused on the technology and making sure it worked. We could have done a more well-trodden genre. We could have avoided historical settings (which would have avoided a lot of headaches around both research and keeping the AI in-period). We could have even adapted an existing story. But, we feel like we learned a lot more by pushing on these boundaries, and had a more interesting end product to show for it.

Focusing on a deliverable, not just on exploration, gave us validation that the methods we explored work, and that AI can be used to tell compelling stories.

Dealing with the problems of a historical setting forced us to find interesting solutions, can be shared with others to give a greater understanding of AI development. 

Starting from our own original narrative made us really focus on what sorts of new experiences are enabled by AI, and led us to create both a game and a story that could not have been done before.

All of this stuff made our lives harder, and it was a significant risk, but we decided to do the things we were excited about and we think it paid off.

We always had a working game – Despite working with novel technology we made sure to create a working game early on. We had a working character you could talk to by week 2. We were doing roleplay playtesting of our entire story to get information on players and see if our story worked by week 3.

Beyond this, we always made sure we had a working build we could test if we needed to. This constant testing and feedback made us catch problems early and allowed us to constantly iterate and improve the game. That ability is essential for all projects, so that unforeseen issues can be caught early, but especially so for those exploring new spaces and needs.

We went with the flow – AI is unpredictable in many ways. Our initial version of the project was using GPT3 for dialogue generation, but when the API for ChatGPT came out half-way through the semester, we decided to switch over to ChatGPT, despite the need to rework a lot of our code and design in the process.But, it was ultimately worth it, as the dialog generation was better and cheaper. On top of that, tweaks to models can come out every few weeks that can change AI behavior, and our team had to be ready to adapt to those changes when they happened. The tech will change on you, and you have to be ready to find out what has changed and adapt to it.

More generally, AI is an unpredictable beast. The outputs it provides can never be fully controlled or predicted, so it is best to work with it rather than fight it.. We held to this philosophy well, whether it be changing the characterization of some of the AI characters to obtain better results from ChatGPT, or embracing some of the ways that players expressed themselves, such as having their character try to do actions or mess with the AI characters in other ways, or even adapting our introduction and wider frame narrative to better match player expectations and what worked with AI storytelling. You are working alongside AI here, and, like any good cooperation, there has to be give and take. Being willing to give up fine-grained control and just accepting good results, even if they are not exactly what you expect, is essential. 

What Went Wrong

We playtested too often – Playtesting is essential, but there are diminishing returns if you do it too often. There was a point late in the semester where we were getting new playtests every day or two, and it was too much. When 90% of the feedback you are getting is things you have heard many times or have known about for weeks, it shows that you should be spending more time on development than continued testing. Playtesting takes time and effort on both the developer’s and tester’s parts, and it is not fair to waste that when you aren’t gaining much from it. It also led to frustration for several members of the team.

We should have had clearer goals for what we were seeking from playtests and been more willing to space out feedback we got so we could act on it in a more constructive way.

We overscoped – With all that we said about the benefits of not taking the easy route, we still overscoped. We had a big list of features that could be used to make the AI better, and we were bad at setting deadlines for when to stop trying to implement them. A bigger focus on polishing the UI and existing systems would have benefitted the game far more in the time we had than adding extra systems for only marginal benefit. It would have resulted in less overall development work as well. We ultimately did shift focus to improvements and more specific parts of character iteration, but it took too long to get to that point. Stronger cut-offs on new features would have benefitted us greatly.

We spent too long on small details – One of the hard things to do when you have a lot of people in a room with good and strong ideas is when to just make a decision and move on. We all struggled with that, whether it was long discussions on character details, or debates on specific wording and phrasing, or minor changes in UI. We got very focused on the minor details at times. We tried to mitigate this by establishing people whose job it was to have the final say, but conversations could still often go on too long and lead to people being confused as to what was actually decided. In a fourteen week project, where time is of the essence, the first solution that works is often good enough, especially when you could spend at least twice the time to only come out with a slightly better result, all at the expense of other parts of the project. 

Conclusion and What’s Next

All in all, we feel that the project was very successful at achieving its initial goals. We are proud of what we made and what we learned throughout this process. There are still things we would like to improve, and there is so much more that could be done given more time, but we ultimately feel that the end product stands on its own and we are happy with it. 

We were also glad to find that not only is AI a viable tool for game development, but it also enables new experiences that could not have existed otherwise. It does not supplant designers, but rather allows them to explore new challenges and express themselves in new ways. We are excited to see what future projects are made with it at the ETC and beyond. 

We have already released the game on Itch.io and are pending approval and final adjustments for a Steam release. We are working on an article targeted at game developers that will go into a lot more detail on the development process and what we learned from it. After that, we will look into submitting it to festivals and giving talks at any conference that we feel has an audience for the game.