AIAuditors is partnering with Carnegie Mellon HCII to aid in the research on fairness in machine learning (ML) by developing a Game With A Purpose (GWAP). The project plans to develop an online game that engages players in (collectively) surfacing and making sense of questionable judgments in AI system(s). It will be released to attract and support everyday users to play and generate usable data as a byproduct of gameplay. Instead of just looking for any mistakes an AI system makes, we are trying to uncover mistakes that will cause harm to various groups of people. We are making it a web-based game in order to increase public accessibility, with both individual and competitive elements, to increase engagement as well as provide a greater variety of data.
The Game
We are creating a two part online card game.
The first part is card creation, where the player tries to come up with statements that are non-toxic, but the AI believes to be toxic.
The second part is the multi-player game. Heavily inspired by games like Apples to Apples and Dixit, each player submits a card and then votes on the cards submitted by the other players, based on the criteria we give them.