Work Preparation:
We discussed with instructors on Wednesday about our halves goal as well as the tasks we want to finish before halves. We went over the three pipelines more in detail and made clarifications so that everyone is on the same page. We also re-visited the topic of failure definition, and our instructors made clear that even though this is an experimental animation project, we are still expected to produce at least one good looking piece that can be shown off on portfolios.
It was also pointed out by instructors that some transition details were missing between scenes in our current storyboard. Many animations are needed in the film, like the Yin and Yang explosion, we need to think carefully how to best express the scene.
Progress Report:
- Finalized the poster and half sheet
- Updated the project website and weekly blogs.
- Updated the storyboard and created an animated storyboard.
- Started greyboxing the clouds and exploring the cloud VFX.
- Continued researching the AI rendering.
- Continued working on the camera angles and the film sequence.
- Started model Pangu’s hand.
Research Results:
- AI Texturing on Cloud Models:
It is challenging to model the clouds with the desired style for this project. We tried to hand model the clouds in .fbx format and apply AI textures onto them. However, the results were not too good.
- Unlike the mountain assets, the AI texture with the clouds is less satisfactory, it seems AI has a hard time understanding the abstract shape of the clouds.
- Like the mountain assets, AI texture tends to skin more colors on the model than asked, it seems not to have the ability to filter the color among all of those textures in its database.
- AI Modeling of Human Body Parts:
AI generation for storyboards and character models has proven very difficult. AI struggles to understand shot composition and character posing when generating storyboards and is only really useful during the initial ideation process. Adobe is working on some new tools to help visualize sketches, but at present they are not public.
AI modeling tools are virtually useless when it comes to the human form, especially if you’re looking for a specific body part. Even if they could get the shape right, the textures and topology are very messy.
- AI Rendering:
We tried to use Stable Diffusion 1.6 + ControlNet v1.1.313 + Ebsynth this week to achieve the Img2img-based animation style transformation.
- Original Feed: Aimed to shoot different head angles (different shading), with mouse open to test if AI could recognize the shape of mouth.
- Following the tutorial and trying with the canny + openpose, turns out openpose doesn’t even work, while at some keyframes the ai recognition is quite inaccurate because the frame recognized by canny is not correct enough. The styles of eyes and the glasses remain quite stable. The clothes we wear are too black which could possibly add a lot of noise to the recognition so that the generation of clothes inside the video keeps flickering.
- We tried to decrease the canny low threshold and the high threshold so that canny could detect the frames more accurately. However, We found that it isn’t always the lower the better. If these thresholds are too low, it would add too many noises to the recognition thus greatly decreasing the results’ quality. So before generating the animation, it’s worthwhile testing the parameters to make sure the canny works the best way it could. And as is shown in the gif, the mouth recognition is well improved since it could generate the tongue and teeth. However, since certain keyframes’ lighting conditions are bad that blur the boundary of the face and neck at that frame, AI couldn’t really tell if it’s the face or neck, which results in those distorted faces, so do these clothes.
Even though we currently only tried with face transformations, we think object transformations within animations might be hard for current situations. Because there are so many models dedicated to human generation while those for environment generation are few, so we might not be able to find one that really could deliver a pleasing result. Also, a model that’s dedicated to a certain theme would work better since the data set it’s trained on is very concentrated that would reduce the noises during generation. This method still needs further experiments, with the help of ControlNet surely could offer some interesting results.
Plan for next week:
- Update the animated storyboard and fill in some scene transition details.
- Playtest and see whether people understand the story.
- Contact Panpan for sound support.
- Continue working on the website and weekly blogs.
- Finalize the camera angles and the film sequence with detailed transitions filled.
- Continue working on the clouds.
- Change plan: iterate on the hand for 1~2 days, and then start to model Pangu full body.
Challenge:
- Hard to estimate how long each approach will take and what potential problems we will meet throughout the semester.
- Hard to estimate the cost for AI tools, and how effective they will be.
- Need to think of better ways to document our research process.
- Need to show all three approaches and their differences on halves, need to convince the faculty that all three films can be completed by final.
- Need to test whether people understand the story.
- Need to make sure we have some powerful shots in the film.
- No sound designer on the team.