License:
Creative Commons Attribution Share-Alike

by Blender Dumbass

Aka: J.Y. Amihud. A Jewish by blood, multifaceted artist with experience in film-making, visual effects, programming, game development, music and more. A philosopher at heart. An activist for freedom and privacy. Anti-Paternalist. A user of Libre Software. Speaking at least 3 human languages. The writer and director of the 2023 film "Moria's Race" and the lead developer of it's game sequel "Dani's Race".

While James Cameron has denied that Avatar 3: Fire and Ash used any generative AI. On January 24, 2026 Joe Letteri ( the visual effect supervisor from Weta Digital ) sat with the Corridor Crew for their weekly show of VFX Artists React where he said that the facial animation for the hero characters for both Avatar 2 and 3 involved a neural network. A neural network? Isn't that a fancy way to say "AI"?

As far as I understand, there is a fundamental limitation with the standard performance-capture used on the first Avatar film. The data gathered by using 3D tracking of points is too noisy. It is good for general layout and some background characters. But it requires a lot of work to tediously clean up before it could be used for something like a face of one of the main characters.

The new approach Weta developed for the second film titled APFS ( Anatomically Plausible Facial System ) tries to solve this issue ( or at least give tools to the artists, to better be able to handle the issue ).

They start by making an anatomically correct model and rig of the actor's face first. This, as you might imagine, is a very hard thing to do correctly. And probably requires a lot of very careful work. Because the next part depends on it being accurate.



This rig, most importantly has a muscle structure that correspond to an anatomically correct muscle structure of the actor. The muscles, as I understand, are not modeled as meshes, but rather as "functions" that distort the face one way or another. Kind of like shape-keys, but with a lot of attention to detail. And probably based on some 3D scanning of the actor in question. For example, some combination of muscles in tension could cause a certain crease to occur that would be hard to capture using standard performance capture methods.



Then, if I understand the system correctly, when the performance is captured, it is captured using 2 cameras, which in combination produce a good 3D approximation of the face in motion.

The custom-built neural network algorithm by Weta then takes this raw 3D motion data and figures out what muscles, on that accurate rig of the actor, should be active, to get the best match for the face. Small creases and everything. And that way they can extract, basically, the muscle twitch data. Which later could be applied on the rig of their character in the film. Which by the way, has to be as detailed ( if not more detailed ) and rigged to the same specifications.

Now, because the characters in the film, for which this technology is used, are most likely Na'vi, the proportions are going to be slightly off. So an artist will need to tweak the muscle twitch data anyway. But because the data is there, and the rig itself behaves more realistically, this drastically simplifies the job of the artist.

If my understanding is correct, in my opinion this "neural network" is not worse than using a denoiser. It is not generative AI. It is a neural network ( which is a fancy way to say "a math algorithm" ) used to convert a raw 3D mesh of an actor, to a muscle twitch data useful for animation. In this particular case ( like with using a denoiser ) I don't see a problem at all.

Happy Hacking!!!

2

Find this post on Mastodon

Find

  Energetic Nova c:0


@blenderdumbass Probably Not the same cause it is using their own assets.

Post