Work Preparation:
This week, we had our half presentation on Wednesday, and we got positive feedback from the faculty. The overall goal and the development process are well accepted and acknowledged by faculty. Thus, we just need to continue our development and iterate on the current two films we have. We did decide to take a stop on approach 2, which is the AI modeling approach, as from previous research results we reached an agreement that AI modeling is not there yet. Later the semester, we would like to conduct more research on AI modeling and its potential; however, we think the model quality it provided is not worth the time developing into the film.
Progress Report:
- Updated the project website and weekly blogs.
- Finished compiling the so-far research results into one documentation.
- Talked with Pan-Pan for sound support.
- Implemented the cloud and mountain assets and VFX into the camera sequence and updated some shots.
- Continued researching AI rendering.
- Continued researching AI modeling.
- Finalized the Pangu’s full body model.
- Tasks and schedule made to prepare for the next milestone.
Research Results:
- AI rendering:
Stable Diffusion 1.6 + ControlNet v1.1.313, Img2img-based animation style transformation, Playblast exported from UE5 style transformation
Based on previous weeks’ experiments, image-based style transformation works the best in terms of our project goal. So, we tried to feed the Playblast made inside UE5 into stable diffusion to help us render the image sequence. The method we use is the same as the one we used in week3, but this time changed from transferring character into environment + character. So, the problem is obvious, how to extinguish the environment from the character.
The solution for now is simple, divide the image sequence into different parts based on different shots and situations:
Based on different scenes each shot is depicting, we also need to adjust the prompt we use to qater the image sequence for better generation. Following are the sample images we pick from each shot to test the parameters and prompt for generation:
Based on these sample images, we generated image sequences for each shot and composed them together.
The problems are obvious: we want black and white, we need to do some color correction; the resolution is too low; we need to denoise. First, we use the same upscaling technique mentioned in week 3 using upsclaer to upscale the frames one by one:
Next, we use Davinci Resolve to do color correction and denoising.
Davinci resolve helped a lot reducing the flickering from frame to frame but there are still spaces to improve consistency.
Image-based style transformation worked well transferring our playblast. However, the current version lacks details and consistency. The best solution for now for increasing consistency is using the denoising method mentioned above, however, for increasing details there are more ways to experiment in the upcoming weeks.
Plan for next week (after fall break) :
- Continue to contact Panpan for sound support.
- Continue working on the website and weekly blogs.
- Start modeling the Pangu face .
- Research more on AI rendering and iterate on rendering the film.
- Work on more cloud VFX and the Yin & Yang debris mesh.
- Animate the background environment of the film.
Challenge:
- Hard to estimate how long each approach will take and what potential problems we will meet throughout the semester.
- Hard to estimate the cost for AI tools, and how effective they will be.
- Need to think of better ways to document our research process.
- Need to make sure we have some powerful shots in the film.
- Need to prepare for the ETC Playtest Day.
- Need to keep Panpan on the same page for sound production.
- AI modeling is proved to fail, and need to shift focus to AI rigging for approach 2.
- People have different plans for fall break, how to make up for the lost time after coming back.