How ILM Made the Ultimate CGI Hulk for ‘The Avengers’
Marvel hoped ILM could create a realistic Hulk for The Avengers in 2012. But it was far from an easy challenge. The previous two films–Ang Lee’s Hulk (2003) and Louis Letterier’s The Incredible Hulk (2008)–misfired with audiences on numerous levels; the main issue is that both films failed to create CGI Hulks with naturalistic appearance or movement.
ILM had played a significant part in that failure. The cartoonish Hulk they created for Ang Lee in 2003 fell below the quality of other CG characters at the time. Hulk’s animation was closer in quality to Shrek than Gollum, the standout full CG character at the time. The Hulk’s giant leaps across vast landscapes compounded this criticism, earning him the unflattering label, the Skipping Hulk. Such criticism had never been leveled at ILM in the past or since. The seed was sown for a comeback, but ILM sat the next film out.
ILM’s failure led to the hiring of another VFX company, Rhythm & Hues, for The Incredible Hulk five years later. There was only one direction Rhythm & Hues could go after ILM’s Hulk bottomed out. Yet the VFX company only made marginal improvements to the jade giant in the second film. Hulk’s animated body and mass had more volume, but he still looked plastic. And he was uglier than Lee’s Hulk with all that sinewy musculature. Overall there was little progress in Hulk’s movement and characterization. He was still a big, poorly animated monster devoid of personality. But at least this Hulk no longer skipped.
In essence, both VFX companies failed to balance Hulk’s strength and body mass with his dexterity. Nor did they balance his explosive anger with a likable performance.
Who is the most realistic-looking Hulk of them all?
In 2012, ILM had a lot to prove and hoped they could wipe the slate clean with the development of the definitive realistic Hulk for The Avengers. Jeff White, ILM’s visual effects supervisor on the film, knew what was at stake, not just for ILM’s reputation but also for the fledgling MCU.
“For The Avengers we knew right from the beginning we could nail New York City and the aliens and the different Avengers. But at the end of the day, this film was going to be judged on the Hulk.”
Of course, we all know the outcome. ILM created a fantastic-looking CGI Hulk, which helped earn The Avengers an Oscar nomination for VFX. While ILM made considerable strides in CG skin, color, weight, facial and motion capture, they also had two secret weapons: director Joss Whedon, a walking encyclopedia of film and pop culture, and Mark Ruffalo, the first actor to play both Dr. Bruce Banner and his alter ego, the Hulk.
Finding the Right Look For Hulk
Once Ruffalo was cast, the VFX team searched through Hulk comic book art for a look that approximated Ruffalo’s facial shape. Like most superheroes, Hulk’s design changed from one comic book artist to the next. The VFX team finally settled on Alex Ross’ comic book illustration of Hulk in the Ultimates, which is one of Marvel’s favorite versions of the character. From there, concept art was drawn up that captured the right balance between Ross’ illustrations and Ruffalo’s facial features.
Comic book art by Alex Ross influenced the look of Hulk in The Avengers.
White told Art of VFX that Whedon was instrumental in the design of Hulk:
“[Whedon] came up to ILM when we were in the trenches of character development on Hulk and had a very in-depth discussion about who this Hulk is, how he moves, literally what his motivation is. He showed us comic book reference of poses he liked. It was enormously helpful for the entire team and the animators at ILM did an incredible job interpreting that into the character on screen. Joss was in our shot reviews with the rest of the great team from Marvel. It was a true collaboration. He was able to identify the big picture items, a shot needed, and let us work through the details.”
Facial Capturing Ruffalo’s Expressions And Morphing Them Into Hulk
With the look of Hulk locked in place, ILM’s job was to merge Mark Ruffalo’s facial features into the CG Hulk. This required the construction of a detailed high-resolution CG model of Ruffalo’s face through facial capturing.
Can you tell the real Mark Ruffalo from his digital double?
Ruffalo sat in Light Stage, an advanced 3D facial capturing technology used for visual effects. Light Stage is a hollow geodesic dome with 300-plus LED lights and multiple cameras placed equidistantly around the structure. The actor sits in the center and the lights and cameras fire capturing a high-resolution model of their head with high-resolution textures.
Ruffalo’s patience allowed ILM to capture a comprehensive expression library, and build a Mark Ruffalo CG model with incredible resolution. ILM even captured inside the actor’s mouth, scanning his gums and teeth. According to Gio Napkil, ILM’s Creative Model Supervisor on The Avengers, the purpose was to capture “the essence of Mark Ruffalo into the Hulk. We took a lot of the skin details, the grays of his hair, even right down to his fingerprints. So we made sure that all of that was in the model just to give it that extra subliminal layer of Mark.”
Mark Ruffalo’s Light Stage facial capture in Zbrush.
This CG model and a laser scan of a urethane cast of Ruffalo’s head were then given to Kris Costa, ILM’s senior creature modeler on The Avengers. Costa used Zbrush to morph and modify Ruffalo’s features into Hulk, effectively retaining the essence of the original actor.
“Once in ZBrush I’m able to use digital brushes to re-create all the details,” said Costa. “So, you know, just like with traditional clay, I’m able to sculpt those wrinkles and all the fine detail around the surrounding areas of the eyes and the lips. And add all the fine pores and wrinkles and scars and things like that.”
Ruffalo’s facial features morphed into Hulk in Zbrush by Kris Costa.
Motion-capture On Set
Mark Ruffalo was very much involved in the action as Hulk on set. A special mo-cap suit and camera rig captured Ruffalo’s performance as Hulk and his interaction with other actors in a scene. This camera rig consisted of four motion-capture HD cameras (two captured his body movements, while two captured his facial expressions).
“[Ruffalo’s mo-cap suit had] this triangle pattern all over it,” explained Kevin Wooley, who worked in ILM’s Performance Capture R&D on The Avengers. “So this new patented suit design works with a new tracking system that likes these corner features of the triangles. So when you see him in the really silly suit, we’re trying to get it so it’s easy to do on set capture as it is to do capture on the stage.”
Once the team captured all of Ruffalo’s face and body movements, this digital double had to be re-targeted into Hulk. The Jade Giant, unlike Ruffalo, “has that huge Hulk brow and he has a much longer distance between the bottom of his nose and the top of his lip,” explained White. “Everything we saw in Mark, we brought into Mark Ruffalo’s digital double and in turn brought to the Hulk and kind of exaggerated a version of that.”
Keyframe Animation For The More Over-the-Top Moments
Even with Ruffalo donning a mo-cap suit, there were just some scenes that he couldn’t do, no matter how willing. Jumping from the Helicarrier onto a jet and pulling the pilot out of the cockpit was just one of the many over-the-top moments that only keyframe animation could achieve.
The fine, realistic detail of keyframe animation worked seamlessly with Ruffalo’s captured performance. So much so, that when you watch the final film it’s near impossible to work out where Ruffalo’s mo-cap work ends and the keyframe animation begins. This is exemplified in the hilarious “puny God” scene where Hulk repeatedly slams Loki into the floor of Tony Stark’s penthouse apartment.
Another challenge for ILM in this scene was getting Hulk to speak. “That shot in particular, with him delivering that line of dialogue [when he says, “puny God”]. We spent a lot of time in animation just tuning the shape of the mouth and tuning the delivery. To get it just right,” said White. The scene involved dialogue lip-sync animation after the motion capture didn’t work. “He looked a little ‘fish-lipped.'” A few tweaks to pixels around his eyelids and eyebrow helped with realism.
Getting Hulk’s Skin Right
A key difference between the first two Hulks and the Hulk in The Avengers is skin detail. Getting Hulk’s skin texture right contributed immensely in making Hulk look like a realistic-looking, green humanoid. Essentially the scans of Ruffalo’s skin in Light Stage became the skin of CGI Hulk. Light Stage uses polarizers which isolate some incredibly fine porous skin texture, giving Hulk that realistic edge.
Light Stage captured Ruffalo’s skin texture down to the pores. Wrinkles and skin details were added in Zbrush before being translated to Hulk.
While Light Stage provided unprecedented detail of Ruffalo’s skin, ILM and Whedon needed to find the appropriate green skin tone for Hulk’s skin to look natural. The previous two films provided valuable lessons on what not to do. The bright lime green of Ang Lee’s Hulk in 2003 was cartoonish and unrealistic, while Letterier’s selection of a grayish-green for The Incredible Hulk (2008) improved appearance but still had a glossy, unnatural look.
For The Avengers, Whedon wanted an even more de-saturated green than Letterier’s grayish-green. “If you are going after skin then going after something more in the realm of skin tones – even if it is green rather than skin color – helps a lot,” said White, who praised Whedon’s choice of green.
Choosing the correct base color is important, but that base color also needs to be corrected in different lighting environments (e.g. under the blue sky or in an interior). To alter Hulk’s base color in different lighting environments, the team de-saturated him down to a gray-scale, balanced him in the shot, and then brought him back to the green shade.
Ultimately Its How Hulk Carries Himself
Of course, creating the right skin tones and facial expressions means little if Hulk does not possess the right weight and movement. Far from being separate issues, weight and movement affected each other. Hulk’s mass determined how he moved. ILM had to determine not only the weight of each movement but also how his muscles moved beneath his skin.
“[ILM] did a lot of animation work in terms of selling the weight,” said White. “And that was a hard slog to get it right and to get all the pieces working together to make his mass believable. Beyond that, we did several rounds of simulation as far as the muscle dynamics and the skin – to help make that all work together.”
The details of Hulk’s movement came down to simulating how muscles and limbs move while in motion on a big bulk like Hulk’s. The final result is pretty naturalistic, and is best seen when Hulk chases Back Widow inside the helicarrier–you can see his muscles jiggling up and down.
Hulk’s musculature responds to different movements, whether running, jumping, lifting or throwing objects, climbing, or pulling himself up. This was achieved with simulations that mimicked the inflation of muscles under strain and flex. The muscles also worked in concert with the skin. Together, subsurface muscle movement combined with the flushing of the skin increased the naturalistic look of Hulk’s body mass.
Overall, ILM’s advancements in CG skin, color, weight, facial and motion capture worked together to create a realistic Hulk in The Avengers. And judging by audience enthusiasm for both Ruffalo’s portrayal and the CGI Hulk, ILM succeeded in delivering the best Hulk thus far. It was perhaps the last time that audiences and critics have been so unanimously praiseworthy of a CGI Hulk. There have been some complaints that the standard has dropped in Avengers: Age of Ultron and Thor: Ragnarok. Either way, ILM delivered and all memory of Ang Lee’s lime green ‘Skipping’ Hulk has been erased. Well almost.
