Hello everyone! We are team AlIluminate, a team of 5 2nd year graduate students at the Entertainment Technology Center of Carnegie Mellon University.
AIlluminate is a project that is producing animation with the aid of AI. The purpose of the project is to integrate AI generative tools into the traditional 3D animation studio pipeline and explore the pros and cons of using AI tools. The team will isolate and test three different approaches to using AI, creating three versions of a film about the creation myth of Pangu, each in a different style. Along the way they will document the process at each stage of development.
Our team:
Jiani Feng – Producer
Ryan Yuan – Artist
Kailai Feng – Artist
Caiyu Zhang – Artist
Steven Goehrig – Artist
Work Preparation:
We introduced ourselves among the team, talked about each other’s strengths and weaknesses, and discussed what we each want to achieve throughout the semester. Based on the internal discussion, we decided to take the more experimental path, to produce one short scene with three different AI approaches and to test out as many AI tools as possible.
We decided on the team’s name “AIlluminate”.
We also met with our instructors, John and Ruth, and reached an agreement on the final deliverable by the end of semester as well as the project goal: to explore the pros and cons of using AI in the animation production pipeline. By the end of the semester, we not only need to present the three films, compare their visuals, but also the processes of the three pipelines. Therefore, documenting our research and implementation processes is crucial.
Progress Report:
- Finished the deliverable goal setting.
- Started to research on the environment concept art, character concept art, AI modeling, AI texturing, and AI rendering.
- Used ChatGPT to generate the script
- Finished hand drawing storyboard scratch from the script
Research Results:
- AI Modeling:
Alpha3D
Works only good on shoes, seems more targeting on commercial/advertisement product applications instead of 3D animation. I think they are using many shoes models fed into the AI so they can have a better result on shoes.
Not good in the animation pipeline especially with our Pangu story.
Meshy
Discord based generative AI, works quite well on generating whatever I want, the result is not precise enough but usable in backgrounds. Can handle different style and different assets, mostly works well in props or individual
With some tuning and time, it might be able to be used in part of the pipeline.
Pixela.ai
Works better with texture generation, not in a desired result for tiling texture, needs to be reworked in substance sampler.
3DFY
Text based generative AI, can generate limited categories: sowards, shields, furnitures, etc.My guess is they are feeding limited categories models to the AI so that they are capable of generating such assets. The text prompt doesn’t really change the result a lot.
Can generate certain types of models, but the mesh/edge flow is bad, textures are bad.
- AI Environment Concept Art:
We mainly used Midjourney to gather different concept art for inspiration.
- We decided that one of the clip styles should be a traditional Chinese ink painting style with a black-gray-white-redish color palette.
- Different AI tools have different keywords to trigger different results.
- Very good for inspiration and iteration.
- Prompt is very important, knowing exactly what is needed and what style if you want a better result (unless you are aiming to brainstorm).
- Both AI tools are being sued for stealing other artists’ works, so there is definitely an ethical issue involved.
- AI Character Concept Art:
This week we began working on the character design for Pangu, the main character of AIlluminate’s film. Within Chinese mythology, Pangu is the first man, and he separated the earth from the sky. He is muscular with ragged hair, and a grass skirt. These details formed the core of the prompts we entered into Adobe’s Firefly tool, and Stability’s DreamStudio tool (both being implementations of their respective AI image generation models).
First, we collected 48 images from Firefly using a couple dozen prompts with only small adjustments between each one, and numerous changes to the style options available in Firefly.
We chose Firefly for the first pass because the application is free to use, and it is trained on imagery that Adobe owns the rights to. My aim was to select designs I thought best met the description while also collecting as diverse an array of options as possible.
After receiving feedback on the best designs from the team, we fed the best options into DreamStudio. This was partly because DreamStudio cost’s money per image generated so we wanted to be more sparing and direct with my usage of it. Stability does not own the rights to all of their training data, but they do at least allow artists to opt out of letting their art be used. It was also for the sake of just trying a different model to get different results. Both models were pretty good at generating asian character designs (although they would default to white people unless otherwise prompted), whereas the teammate struggled to get good asian designs out of Midjourney.
Importantly, DreamStudio offers a fairly robust suite of editing tools. This makes it easier to edit the photos myself and submit them back into the model. Firefly also has decent editing tools, but does not support feeding custom images into the model this time. The ability to edit and feed images back in and adjust how much the prompt or image was used to generate new designs was integral to getting the results we were looking for. We would go as far as to say that good digital art skills are integral to utilizing image generation models effectively, as they cannot be relied upon to produce a consistent and accurate final product.
- AI Rendering:
We also researched on the AI rendering tool. We used Stable Diffusion 1.6 + ControlNet v1.1.313. We used img2img for style transformation & ControlNet for stability. Inputting the original video, we transferred it in different ways to test the video stability.
Test out a different style.
Add color to the video.
Try with the upscaler function script, it turns out to have more noises.
Plan for next week:
- Continue researching
- Discuss the final deliverable goal with instructors
- Finalize the styles we want for three films
- Prepare for the quarters presentation
- Test out some of the AI tools
- Team bond dinner
- Team Logo
Challenge:
- Hard to estimate how long each approach will take and what potential problems we will meet throughout the semester.
- Hard to estimate the cost for AI tools, and how effective they will be.
- Need to convey clearly to instructors what our deliverable goal is.
- Need to visualize the storyline with a detailed storyboard and settle down the camera angles.