James Cameron and Peter Jackson are the kings of the CGI world. Cameron, of course, directed Titanic, the highest-grossing movie of all time—which he says he’d make with no ship if he were filming today. Jackson was the guy behind bringing Middle-earth to the big screen in the Lord of the Rings trilogy. Now they are back with Avatar and The Lovely Bones, two of the most-hyped films of the holiday season. Newsweek asked them about their new films and how technology is changing Hollywood. An excerpt of the transcript is printed below:

Cameron: So how’s the road trip been on The Lovely Bones?

Jackson: It’s all right. Not too bad. Having a harder job getting over the jet lag than I normally do, but never mind. Getting older, I guess. I’m in … Berlin.

Cameron: Ha, ha! You had to think about it for a minute!

Jackson: I did! I’m flying to Paris as soon as this phone call is over. So we’re talking about technology and movies?

Cameron: People often ask us about the future of filmmaking because we’ve both been innovators in the last few years, creating cutting-edge stuff that gets widely or narrowly adopted. I think the simple answer is that filmmaking is not going to ever fundamentally change. It’s about storytelling. It’s about humans playing humans. It’s about close-ups of actors. It’s about those actors somehow saying the words and playing the moment in a way that gets in contact with the audience’s hearts. I don’t think that changes. I don’t think that’s changed in the last century.

Jackson: There’s no doubt that the industry is in a weird position. It’s not just Hollywood—it’s international. The loss of the independent distribution companies and the finance companies, and the lack of ability to get medium-budget films these days. The studios have found comfort in these enormous movies. The big-budget blockbuster is becoming one of the most dependable forms of filmmaking. It was only three or four years ago when there was a significant risk with that kind of film. Now, especially last summer, we saw blockbuster after blockbuster be released, and they all had significant budgets and they’re all doing fine. It almost doesn’t matter if the film is a good film or a bad film, they’re all doing OK. They’ve lost the ability to have that happen with a low-budget movie and with midrange-budget movies.

Cameron: But they’ve also lost the courage to make, frankly, a movie like Avatar, which is a blockbuster-scaled movie not based on prior arc. All the blockbusters of the last four years, like Transformers, Harry Potter, Spider-Man—they’re all films based on other films or part of a franchise. The idea of making a film of that scale that’s a unique piece has been lost. In the meantime, we have all these increases in technology. And there’s no clear way to pay for these blockbuster movies in the old traditional way. It’s not clear that the technology will come down in price in the near future.

Jackson: People are holding on to the idea of lowering the price. The vast majority of the CGI budget is labor. Unless everything goes to China or Eastern Europe in the sweatshops, that sort of approach, labor is never going to go down. It’s only going to go up.

Cameron: Because computers don’t create beautiful images. People do. Down at your place in Wellington [New Zealand], we had 800 people working on Avatar for the past six months.

Jackson: The ones that are conscious anyway.

Cameron: I’m sure there was a big night at the Wellington pubs a few days ago when they turned over their last shot.

Jackson: I think there were a few pillows and sleeping bags under desks. A lot of media attention is switching to technology in the wrong way. They’re saying the industry is in trouble; will 3-D save it? That really doesn’t have anything to do with it. The industry is in trouble, but it has nothing to do with technology, nor is technology going to necessarily be the savior.

Cameron: No, it can’t: 3-D may help define the idea of the big show at the cinema, the cinematic experience, but I think the heart of the cinematic experience is the group experience. It’s the psychology of sitting in a dark room with a bunch of people and reacting to something, and feeling like your reaction is the same as the rest of the group, a way of proof-checking your emotions are normal.

Jackson: Or not.

Cameron: If you’re the one guy laughing out of 400 people, you’re obviously out of step. I don’t think that’s going to change. People have been downloading films, watching films on laptops, watching films on iPods for quite some time now. Ticket sales are not dropping at the same rate that those other methods of media are rising. I came to filmmaking in the early ’80s, and it was a time of deep economic recession. It was a time when VHS home video was taking money from the theaters. The film industry was depressed. That’s what I knew—a state of upheaval and change. It all sorted itself out. These things always sort themselves out. The fundamental question is: is cinema staying or is it going away? I think it shows no signs of going away. I feel quite confident you and I are going to make the kinds of films we love 10 and 20 years from now.

Jackson: I do too. In addition to the theatrical experience, we will be seeing a lot of other forms of distribution and delivery, which is going to be interesting. We have things like Xbox Live with all the subscribers. It’s not going to be too much longer before Xbox Live produces programming. There are so many opportunities there. Everybody is playing a defensive game. Nobody is going on the attack and being brave and courageous, apart from you.

Cameron: They always say pioneers are those guys flying on planes with arrows in their backs. 3-D will find its place. It’s like color. Color didn’t affect the career of a single actor. And then people will find out about the intimacy to 3-D that can add to a dramatic film that’s not even on the radar of the Hollywood studios right now.

Jackson: I find personally that within 10 minutes I forget that it’s in 3-D, in a good way. The only thing about 3-D is the dullness of the image. But that’s a relatively simple technical hurdle to overcome. It’s just brightening the image.

Cameron: It’s already been overcome. The new technology has already solved the light-level problem. We think it looks fantastic.

Jackson: How far away are we from taking glasses out of the 3-D equation?

Cameron: I’ve seen displays at a laptop size and a relatively modest plasma size that work quite well. You have to situate your head to the sweet spot so you don’t get the double image. But people are always turning the laptop display to get the best image. I can imagine three or four years from now an iPhone that’s 3-D-enabled that doesn’t require glasses that you can watch a movie on. Certainly laptops will be here before that. I think the ones that succeed in the marketplace are the ones that initially make their sets, their displays, to be able to use the glasses. If you’re going to do a Super Bowl party you’re going to have a bowl of glasses on your coffee table, and they’re going to be the disposable kind. And then eventually I think the glasses have to go away for home use. I think that will happen within five years.

Jackson: I’m seeing there’s a lot of misunderstanding about motion capture at the moment.

Cameron: The irony is that one of the first examples of motion capture that worked so beautifully is Gollum in the second and third of your films. Suddenly this new idea had burst on the scene, that a quasi-human creature could be created with such heart and soul.

Jackson: With Gollum and Kong, the key thing that we did was the eyes. I think Gollum and Kong represented the best eyes that I’ve seen in a CGI film.

Cameron: The experience of creating a soulful performance is through the eyes: knowing how to rig eyes, how to light for eyes, get the reflections and refractions in the eyes. Of course, we had big-eyed characters, which we did on purpose. We couldn’t accomplish the character we’re doing in Avatar through any kind of makeup means. That’s been explored for 30 years of Star Trek and Star Wars. But I think the thing I hope that the media can convey to audiences is that this is an actor-driven process. Nayteri, in my film, for example—she is what Zoe [Saldana] created 100 percent. Initially I thought we want to keep the technique under wraps. We don’t want to pull the curtain aside and show people how we’ve done this; we just want to show you my magic. But I’ve recently changed my tune. I want people to see a side-by-side image of Nayteri in a scene and Zoe doing the scene, so they understand that it’s a physical and facial performance. Zoe took months of training at archery and martial arts, so she could move a certain way and have a certain grace. It’s something she created that just translated to her character. This is a highly actor-driven process.

Jackson: Actors will never be replaced. The thought that somehow a computer version of a character is going to be something people prefer to look at is a ludicrous idea. It’s just paranoia. What is great, when you would have used prosthetic makeup, you have motion capture to do a more emotive version. That’s great for nonhuman characters, but in terms of creating nonhuman beings—why on earth would anyone want to do that? It’s so expensive. It’s 20 times more than an actor’s going to cost.

Cameron: The other thing that people aren’t talking about, you can take an actor of a given age, and you can transform their age. Additive makeup can age somebody, but it’s hard to make someone younger. Let’s say you have a novelistic storyline where you cast an actor in their 40s, but the first time you see them they’re 15 years old and the last time you see them they’re 80. This is the Benjamin Button idea. Clint Eastwood could do another Dirty Harry movie and look the way he looked in the ’70s. He would still be making all the performance choices. It would be his voice. We’d just make him 30 years younger. If I did Titanic today, I’d do it very differently. There wouldn’t be a 750-foot-long set. There would be small set pieces integrated into a large CGI set. I wouldn’t have to wait seven days to get the perfect sunset for the kiss scene. We’d shoot it in front of a green screen, and we’d choose our sunset.

Jackson: There are all great tools that people haven’t quite gotten their heads around yet. But one of the things that has happened [is that] people focus on technology. Probably the film industry has been guilty; there’s more attention spent on the technical aspects than the story. That’s led to a self-fulfilling prophecy. People regard CGI as a gimmick, they almost blame CGI for a bad story or a bad script. They talk about CGI as if it’s responsible for a drop in standards. We’ve gotten to a point now where there isn’t nothing else we haven’t seen. We’ve seen dinosaurs, we’ve seen aliens; with Avatar we’ve seen realistic creatures. I think we’re going to enter a phase where there’s less interest in the CGI and there’s a demand for story again. I think we’ve dropped the ball a little bit on stories for the sake of the amazing toys that we’ve played with.

Cameron: I think you’re right. What’s interesting in the marketing evolution of Avatar is that we put out a teaser trailer that was all about the imagery, and people were less than satisfied, because they weren’t learning enough about the story. We put out a story trailer that set the stage and told you what the main character was, and all of a sudden people were wildly excited about the movie. There’s the proof within the marketing evolution of a single film.

Director James Cameron had many reasons to be happy the morning that this year’s Oscar nominations were announced: His blockbuster movie “Avatar” tied for the most with nine, including best picture and best director. But he was dismayed that his cast, including stars Zoe Saldana, Sam Worthington and Sigourney Weaver, was shut out.

In fact, unlike the great majority of best picture nominees, the “Avatar” actors have not nabbed one major critic’s award, or guild prize. The snubs reflect the apparent ambivalence of the film community — especially actors — to “Avatar” and its revolutionary use of “performance capture,” a new technology that combines human actors with computer-generated animation to create the blue, 10-foot-tall creatures who are the heart of the movie.

To the uninitiated, it raises basic questions: Is this acting, or is it animation? And, does this suggest that actors could become obsolete? It’s an issue that provokes a strong response from Hollywood figures, from best actor nominees Jeff Bridges and Jeremy Renner to directors Cameron and Steven Spielberg.

“I’m sure they could do it now if they wanted. Actors will kind of be a thing of the past,” Bridges told Tribune Newspapers the day nominations were announced. “We’ll be turned into combinations. A director will be able to say, ‘I want 60 percent Clooney; give me 10 percent Bridges; and throw some Charles Bronson in there.’ They’ll come up with a new guy who will look like nobody who has ever lived and that person or thing will be huge,” he said.

Renner, nominated for ” The Hurt Locker,” put it this way: “Some movies are actors’ kind of movies and some movies are more directors’ movies. ‘Avatar’ is a spectacle. It’s a beautiful experience, but it’s not really an actors’ kind of movie. It doesn’t really allow for an actor to truly tell a story. The director’s telling the story in that one.”

Perhaps mindful that actors make up the largest Oscar voting bloc, Cameron fiercely promotes the contributions of his cast to the success of “Avatar.” He and other advocates of performance capture (known as “motion capture” in its previous, less sophisticated incarnation), including Spielberg, say not enough actors have experienced the process to appreciate it.

“There’s a learning curve for the acting community, and they’re not up to speed yet,” Cameron said. “We didn’t get out and proselytize with the Screen Actors Guild as we probably should have to raise awareness. Not only should they not be afraid of it, they should be excited about it. There is a new set of possibilities, after a century of doing movie acting in the same way.”

Cameron describes it as “an actor-driven process.”

“I’m not interested in being an animator. … That’s what Pixar does. What I do is talk to actors. ‘Here’s a scene. Let’s see what you can come up with,’ and when I walk away at the end of the day, it’s done in my mind. In the actor’s mind, it’s done. There may be a whole team of animators to make sure what we’ve done is preserved, but that’s their problem. Their job is to use the actor’s performance as an absolute template without variance for what comes out the other end.”

“I like to think of it as digital makeup, not augmented animation,” said Spielberg, who is using Cameron’s “Avatar” technology in his new movie, “The Adventures of Tintin: The Secret of the Unicorn.” “It’s basically the actual performance of the actual actor, and what you’re simply experiencing is makeup.”

In the case of “Avatar,” Spielberg said, “the digital makeup is so thin you actually see everything that Zoe (Saldana) is doing. Every nuance of that performance comes through digitally.”

Spielberg and Cameron say that making a movie in performance capture is, for the actors, very similar to performing a play.

“Motion capture brings the director back to a kind of intimacy that actors and directors only know when they’re working in live theater,” Spielberg said.

Recording takes place on a spare motion-capture stage called the volume. Actors wear skin-tight bodysuits with reflective markers; every movement is tracked by an array of more than 100 fixed cameras. A specialized head-rig camera records the actor’s face and eyes.

“The virtual camera is always active,” explained “Avatar” producer Jon Landau. Gone is the need for camera and lighting setups, makeup retouches and costume fittings. Scenes do not need to be shot repeatedly from different camera angles. Instead, the camera data are fed into a computer that creates a 3-D replica of the actor’s every movement, and the director can just add his camera moves — from any perspective — digitally.

“There’s a purity to it. You can’t rely on anything else but your own skill as an actor; (it) enables the actor to shoot the scene in one take without worrying where the camera is,” said Andy Serkis, a veteran British stage actor who pioneered motion-capture acting as Gollum in Peter Jackson’s “Lord of the Rings” trilogy. Serkis also took the title role in Jackson’s remake of “King Kong” and is performing in Spielberg’s “Tintin.”

“If you don’t have the performance, the rest is dressing,” Serkis said. “You can’t enhance a bad performance with animation. You can’t dial it up, lift the lip or the eyebrow. It has to be right at the core moment. It’s the same as conventional shooting.” For actors to not recognize “performance capture as acting is bad and disrespectful. It’s also Luddite.”

In the case of “Avatar,” some complain that Cameron’s characters are too one-dimensional to merit their actors a nomination, but others believe that “Avatar” star Saldana, whose every minute on screen is in performance capture, was robbed of recognition.

“Zoe played Neytiri with such strength, grace and force. If the audience realized just how much, they would have appreciated the performance more,” said “Avatar” co-star Sigourney Weaver. “The technology is so innovative, and it will just continue to get more innovative. We might as well recognize (the contributions of actors) now.”

From a director’s standpoint, recording in performance capture is unusually free and fast. On a typical day of a live-action production, a director might complete a dozen or so scenes in which the lights, cameras, scenery and actors are repositioned. Spielberg said that on “Tintin,” he completed 75 setups a day on the motion-capture stage and finished principal photography in 30 days. That’s less than half the time it would have taken to shoot a live-action version of the film.

“It allows the director and cast to focus on the performance,” Spielberg said. “The director sits right on the floor (with the actors). Because he’s not wearing a motion-capture suit, he appears invisible.”

“One hundred percent of my focus is on the actors,” Cameron said. “I’m not thinking about the lighting, the dolly, or waiting around … to light the shot.”

Though performance-capture veterans speak enthusiastically about the technique, questions remain. Many wonder whether Saldana will get the kind of career boost usually associated with co-starring in a box-office bonanza. The Screen Actors Guild recently appointed a committee to look into what SAG President Ken Howard described as “pay and recognition” issues associated with performance capture in movies and video games. In fact, studios haven’t formally recognized SAG’s jurisdiction over the work, leaving it up to each employer to decide whether the performers receive standard union benefits such as minimum pay or meal breaks.

Moreover, the actors are not the only ones unsure about their primacy in the process. There’s also a branch of animators who don’t want their contributions overlooked. Cameron points out that it took a team of 20 or more animators at the Weta Workshop in New Zealand nine months to fully animate each “Avatar” character.

“The academy has to come to terms with where (performance capture) goes,” said director Henry Selick, whose ” Coraline” is nominated for best animated film. “Is it animation? Is it a new category? I’m like the academy. I don’t know where it fits. I will tell you this: Animators have to work very, very hard with the motion-capture data. After the performance is captured, it’s not just plugged into the computer which spits out big blue people. It’s a hybrid.”

Tribune Newspapers writers Richard Verrier, Amy Kaufman and Yvonne Villarreal contributed to this report.

http://www.chicagotribune.com/entertainment/sc-ent-0224-avatar-actors-20100224,0,1272757.story

By: Barbara Robertson

When we think about the first films to convince directors that visual effects created with computer graphics could open their imaginations, two films immediately come to mind: James Cameron’s The Abyss, in which a transparent CG character communicated with an actor, and Cameron’s Terminator 2, which starred a digital, liquid terminator and is lauded as the first movie to show the power of a digital pipeline. Both films won visual effects Oscars, as did Cameron’s Alien before, and Titanic after. Titanic, released in 1997, still holds the record for the largest box-office revenue: $1.8 billion. It was the last feature film Cameron had made. Until now.

A new facial motion-capture system devised by Weta Digital captured actor Zoe Saldana’s
facial expressions and mouth movements to help animators give Neytiri, a CG character, an
emotional performance. The long-awaited and highly anticipated Avatar, written, directed, and produced by Cameron and released by Twentieth Century Fox, pushes digital filmmaking into new worlds. It will immerse audiences in an alien environment, one created entirely with computer graphics and projected, in theaters so equipped, in stereo 3D. Cameron used a Pace Fusion 3D camera to film the live-action segments, but they comprise a small percentage of the film. Weta Digital created the alien planet Pandora and the CG characters and creatures that inhabit it, animating the characters using data from actors’ performances on motion-capture sets. Will it have the same impact on visual effects as did Cameron’s earlier films?

“It certainly changed the way we do things,” says Joe Letteri, senior visual effects supervisor at Weta Digital. “We had to go through a complete re-tooling and re-architecting.” Now a partner at Weta, Letteri has won visual effects Oscars for two episodes of The Lord of the Rings and for King Kong, along with an Oscar nomination for his work on I, Robot while at the New Zealand studio.

In particular, Letteri notes, the studio revamped systems for real-time facial motion capture and muscles, created methods for growing a rain forest in which most of the movie takes place, implemented new lighting techniques, built a compositing pipeline to handle stereo 3D, and more. “We could not allow ourselves to cheat anything,” he says. “Everything had to be done correctly; there was no place to hide.”

Weta used an absorption-based subsurface scattering routine to give the blue-skinned avatars and Na’vi a fleshy, believable look.

In the film, Jake Sully (actor Sam Worthington), a paraplegic war veteran, is given the opportunity to inhabit the athletic body of an avatar. He opts in. His avatar is an alien, a Na’vi, a race of humanoids that populate the planet Pandora. He, like all Na’vi, is blue. A 10-foot-tall biped with a stretched, cat-like body. Almond-shaped eyes. Tail. Pointed ears. Through his avatar, Jake immigrates to Pandora, a lush planet filled with waterfalls, jungles, and six-legged creatures, some of which fly. There he meets the beautiful Neytiri (actor Zoe Saldana) and assimilates into the Na’vian culture.

Everything on Pandora—every plant, creature, and character—is digital, created by artists using computer graphics tools and moved by animators working with keyframe and motion-capture data.

“The planet was really inspired by Jim’s [Cameron] underwater dives,” Letteri says. “There’s bioluminescence. The creatures have blue skin, and the animals have vivid patterns. We all know the rules: Big animals don’t have vivid colors. But, they do underwater, and Jim said they can exist on this planet. So we brought that color palette to the surface and made it believable. However, the big thing was that Jim wanted to do facial motion capture.”

Performing Characters

For Gollum in The Lord of the Rings, Weta had captured Andy Serkis’s body, not his face. For King Kong, they glued markers on Serkis’s face and captured him in a high-resolution volume, and then retargeted the motion data to Kong’s CG face. “Jim didn’t want to go that route,” Letteri says. “He was more interested in a video head rig.”

To make a head-mounted system that would encumber the actors as little as possible, Weta decided to create software that could track facial movements using one camera. Then they took it a step further by re-projecting the motion onto a 3D model in real time.

“We knew Jim would have real-time motion capture on the stage for the characters, and would be recording the faces,” Letteri says. “We thought, wouldn’t it be cool if we could do real-time faces? We knew he was coming in six weeks, so we did some all-nighters and got a system working.” When Cameron arrived, he could see actors on stage wearing a head rig that was driving the facial expressions for a CG character in real time.

Stephen Rosenbaum—who had been on the crew at Industrial Light & Magic for The Abyss as a CG artist, was a CG animator on Terminator 2, and who had won a visual effects Oscar for Forest Gump—was the liaison between Cameron and his Lightstorm group in Los Angeles and Weta in New Zealand. He helped integrate Weta’s creatures, avatar puppets, and facial-capture system into previs and the real-time motion systems developed by Lightstorm and Giant Studios. Rosenbaum was one of six visual effects supervisors at Weta who worked with Letteri on the film. The other five were Dan Lemmon, Eric Saindon, Wayne Stables, Chris White, and Guy Williams.

“Lightstorm created environments at a previs level,” Rosenbaum explains. “We created the creatures and character puppets at Weta that they used within the environments. Giant used our puppets during motion capture. And, when they had scenes where actors needed to interact with creatures, we also provided pre-animated characters so they could see the action during motion capture.”

Giant and Lightstorm performed the real-time motion capture that allowed Cameron to see the CG version of the film at a game-quality level as the actors performed in a motion-capture volume approximately 40 feet wide by 70 feet long. Giant set up the volume using close to 120 industrial cameras from Basler Vision, and handled the re-targeting, in real time, of motion from actors onto the rendered, 10-foot-tall aliens. Lightstorm’s virtual cinematography system, developed by Glen Derry, blended the characters into the virtual set using Autodesk’s MotionBuilder for real-time rendering.

“We could tie into the body capture and add our facial capture simultaneously,” Rosenbaum says. “So [Cameron] could see the body performance and the facial gestures happen [on the CG characters] with the dialog, which was a nizce feature.”

The real-time facial performances weren’t always practical—video projection onto the characters’ faces was sufficient for all but the most subtle scenes. However, Letteri believes it’s game changing.

Weta modeled all the plants in the rain forest on Pandora, seen here virtually, using a rule-based growth system. Some plants have as many as one million polygons.

“It’s one of those things,” Letteri says. “You can see a motion-capture demo, and it’s kind of interesting. But, on set, seeing actors and CG characters performing at the same time, well, that’s really cool. It doesn’t even demo well in a video. When you’re there, it’s a whole different feeling. You have to see it in person.”

Rosenbaum estimates that more than 80 percent of the film is virtual. “We’re delivering about 110 minutes of full CG,” he points out. “I would guess that another 20 minutes have a combination of CG and live action. And, there are some other VFX facilities helping out. We sent some flying creatures, Na’vi, environments, and vehicles to ILM, Framestore, and a few other vendors, as well. But, the bulk of the CG work is being done at Weta.” The list of other vendors that worked on previs and postvis for the film includes BUF, Halon, Hybride, Hydraulx, Lola, Pixel Liberation Front, Stan Winston Studio (now Legacy Effects), and The Third Floor.

Capturing Faces

Each actor captured on set wore a helmet with a lipstick camera attached to a boom arm, and green makeup dots on his or her face. The crew positioned the camera between the actor’s nose and upper lip to capture the mouth movement and to see the eyes. To paint the dots, the makeup artists used a vacuform mask cut with small holes designed for each actor. “We’d put the mask on the face, draw a pen mark for the dots, pull it away, and paint on the green dots,” Rosenbaum says. “The actors loved it. It took only five or 10 minutes and they were back on stage.”

To plot the dot pattern, the facial motion-capture crew had first taken video of the actors doing a FACS session—creating particular expressions, mouthing phonemes, doing prescribed facial gestures—and, if they had dialog, saying their lines. The FACS analysis helped the crew identify major muscle groups for each face so they could position the dots, sometimes as many as 70, most effectively.

For the eyes, Weta developed software to track the pupils. “We had an LED array around the camera so we could illuminate the face and see the pupil clearly,” Rosenbaum says. “And if we couldn’t get good data, we’d track the pupils from the video. Traditional facial capture has always been a problem, but I think our eye movement is fantastic. It sells the characters.”

The eye movement was particularly important because although the avatars have eyebrows, the Na’vi didn’t, so their eyes needed to express much of their emotion. Yet, the iris in the Na’vi eyes was so big, the white of their eyes showed only when they were shocked.

“We ended up adding a stripe pattern to suggest eyebrows,” says Andy Jones, animation director. “We studied Zoe’s [Saldana] expression, and found it was really tricky to get the same feeling on her CG character without eyebrows. To prove it to [Cameron], I roto’d Zoe’s eyebrows out of her face, and he realized what we were up against. That’s when we textured in a pattern to get the feeling of eyebrows back in there.”

The motion captured from the actors on stage drove a facial system developed by Jeff Unay on their corresponding CG characters. To help with the lip sync, character designers had created the lips on the Na’vi to match those of the actors performing them. “We kept the characteristics of the actors and reshaped them into alien characters,” Letteri says. “That gave us a good basis.”

“Solving” software applied the data to Weta’s facial system, and a facial-solving team adjusted the result. The motion data worked best for lip sync and mouth movement; animators spent more time tweaking brow and eye animation. “When the overall expression straight out of the facial solve was not what it should have been, the team would push the data around to get the right poses and extremes, yet still keep the live feeling of the data,” Jones says. “As the team adjusted poses with sliders—they called it ‘tuning’ because they tuned the solve on various frames—the solving software learned which poses to use.”

Unay based the underlying system on blendshapes. “We started with a dynamic muscle rig for the faces, but although it was good at preserving volume, it was coming up short in terms of level of detail,” Jones says. “[Cameron] was very specific. If he saw tension in Zoe’s mouth, he wanted exactly that [in Neytiri]. We had to art-direct and sculpt her face.”

So, Unay modeled blendshapes to mimic a volume-based system using FACS, which describes the muscle groups that control parts of the face. Thousands of shapes. The resulting rig for Neytiri, for example, has 1500 blendshapes. “The animators use sliders that control only about 50 shapes at a time,” Jones says. “The system switches to banks of shapes depending on which muscle sliders they move. It all happens under the hood without the animators knowing. The combinations of shapes look amazing; the skin looks like it’s pressing and pulling.”

As the animators worked in Autodesk’s Maya, they could bring up, on their screens, reference video shot in HD from multiple angles. “We could see the skin and get the timing from the helmet camera, but it distorted the face too much to see the overall mood,” Jones says. “We needed cameras farther away.”

Animators at Weta persuaded director James Cameron to add a stripe pattern to suggest eyebrows on the Na’vi’s faces to help give the computer-generated characters the same emotional feeling as the actors performing them.

Animating Performances

Animators also keyframed Na’vi ears and tails. “We’d whip their tails around if they were upset, and use them as a counterbalance when they ran,” says Jones. “They were like another appendage. We also found the ears really useful for adding emotion to the character.” The ears tell when a Na’vi is angry or shocked, just as they do for cats and dogs.

For the Na’vi bodies, the motion capture worked extremely well. “Giant’s body capture was fantastic,” Jones says. “We still had to animate their hands and fingers, but the offsets and targeting and retargeting was well done. They kept the weight. And, the data was clean.”

The characters’ design might have helped with the retargeting. Rather than completely altering the human proportions, the designers created the Na’vi with similar proportions to humans, but with slim hips, narrow shoulders, and long necks. “It made the retargeting process easier,” Jones says.
Oddly, although animators often use motion-captured data to add the tiny movements to help bring alive a character that is standing still, Weta’s animators found themselves adding jitter to the mocapped data in some cases.

“When someone was yelling or screaming, the high-frequency jitters were often filtered out,” Jones explains. “The system couldn’t distinguish between muscle shake and noise precision. So we would animate it back in, and all of a sudden it felt like the characters were screaming, not just opening their mouths. We had the body muscle rig, but when a bicep fires, there needs to be a jitter. When [Cameron] saw us doing that, he really loved it.”

The muscle rig is new, developed at Weta specifically for this film. “It’s a dynamic system that simulates muscles properly,” Saindon says. “It calculates the fat layers and colliding volumes much more accurately than in the past.”

Prior to this, after animation, the character TDs needed to fine-tune the look of the character and fix problems—intersections, muscles that didn’t look right, and so forth—by sculpting the character on a shot-by-shot basis. With the new system, that was rarely necessary.

“We’d get something much more accurate and realistic straight out of the box,” Saindon says. “We had to do little in the way of going back and fixing things.”

Creatures

In addition to the characters, Weta animators performed approximately 10 creatures, a hellfire wasp, and thousands of insects. “Every single frame has something alive in it, whether it’s a moving plant or bugs,” Williams says.

Of the creatures, four fly and most have six legs. “Our first approach was typically to hide the middle legs, animate the animals as quadrupeds, and then bring the middle legs back in,” Jones says. The animators might animate a horse-like creature by having the leg movement cascade, or change the gait by changing the offset. A cat-like creature might arch its back, lift its front legs, and use them as arms and hands.

Jake learns to ride a creature that looks like a flying horse, and for those shots, the crew used a gimbaled motion-control rig. “The good thing about motion capture was that it gave us the posing [Cameron] liked for the character on top of the creature, where the character should be looking, and the riding style,” Jones explains. “But it was obvious that his legs weren’t reacting to his chest popping up and down, so we couldn’t use the motion capture completely.”

Am I Blue?

Facial capture was perhaps the biggest challenge. The second biggest challenge for the technical team was keeping the aliens from looking like someone had poured blue paint on them. “It was a tricky problem,” Letteri says. “They needed to have warmth under their skin, so we had to find the right shades of blue and blood color that would look good in firelight, blazing sun, overcast skies, and rain. Blue skin quickly wants to look like plastic.”

For skin texture reference, the crew did photo shoots under controlled lights of young people with the most perfect skin they could find. “We discovered that even someone with nearly flawless skin still has lots of imperfections in displacement and color. They have nodules, bumps, pink around their eyes, and blotchy layers,” Williams says. Painters added these imperfections to the texture maps and created a pore structure for the aliens that looked realistic. All this helped make their skin come alive.

As for the color, even though the aliens had blue skin, the crew put red blood in their veins, and did so without turning their skin purple. “Before, we had more of an analytical approximation for subsurface scattering,” Williams says. “We went to an absorption-based subsurface scattering routine. The system we use now does proper frequency-based scattering.”

Because they used the actual wavelength for red transmission through the nose, ears, and pores of the skin, the red blood didn’t cause the blue skin to turn purple. They also added a little red to the skin tone. Then, they applied some of the same techniques and shaders written in Pixar’s RenderMan to the plants.

Deep in the Jungle

“We cross-pollinated the efforts,” Williams says. “The plant shader now uses the skin shader.” The plants, however, aren’t blue, even though they started that way. Blue light from a blue sky bouncing off blue plants onto blue-skinned characters created uninteresting images.

“We needed to have other colors hitting the characters’ skin to give them the kind of complexity that helps make them look real,” Williams says.

At night, as the characters walk through the jungle, the plants glimmer with bioluminescence. The CG artists used subsurface scattering to cause thick plants to glow like a wax candle. “Some plants just have a glowing moss over them,” Saindon says. “It depended on the plant and how [Cameron] felt it should look.”

To create the rain forest, the Weta artists started with FBX files from Lightstorm that they imported into Maya scenes. “We had simple representations for where the trees and plants were,” Saindon says. “Jim moved and placed things where he wanted for camera angles. So, we did a one-to-one match at first to get a layout that he specifically liked.”

Because the plants needed to be dynamic, all of them are models created using a rule-based growth system. Although they average 100,000 polygons, some have as many as one million polygons.
“The plant-growing tools were almost like a modeling tool,” Williams says. “Once we grew a plant, we could instantly create variants by changing the seed value for the random functions.” The variants might change the number of branches and sub-branches, the height, the silhouette, the age, or other parameters.

The crew planted the jungle using painting techniques to place trees, shrubs, and grass. “It’s similar to [Maya’s] Paint Effects, but we aren’t creating geometry,” Saindon says. “The system is taking pre-existing geometry and placing actual full-res models at correct angles on the ground.”

They also used Massive’s software to grow forests. When artists planted seeds on a terrain, Massive would simulate a forest growing and competing for light and space. Bigger trees grew quickly, smaller plants died, and shade-loving ferns grew around the base of the large trees.

Jake Sully (actor Sam Worthington) prepares to inhabit the avatar body resembling a Na’vi, seen forming in the tank behind. The color palette for the film reflects James Cameron’s fascination with the underwater world.

“We’d create large areas, and then on a shot-by-shot basis, would sculpt scenes to play well for the camera and the depth of the scene,” Williams explains. “All of our show is done inside Maya, and everything in the jungle is 3D, so when you move the camera around in Maya, you get a real 3D sense.”

To light and render the massive jungle, Weta implemented two techniques: stochastic pruning and spherical harmonics. The stochastic pruning threw away unnecessary geometry on the fly as a plant moved away from camera. “It might take a fern with a million polygons and push it back to a few pixels when it’s in the distance,” Saindon says.

Spherical harmonics, a technique used for real-time rendering in video games, made it possible to light the rain forest. “Basically, we store coefficients for angles,” Saindon says. “We calculate the harmonics for each individual plant, all the lighting angles, and store that on the geometry. That allows us to drop simple lights into the scene and still get proper occlusion from each plant. The plant does its own self-occlusion using its own harmonics, seeing what should be occluding what, and stores the information. That means we can light an entire jungle with one light. We could get complex lighting with a very simple setup. We couldn’t have done the movie without it.”

Even so, the data processing requirements for the show were enormous. In addition to the characters, Weta created volumetric explosions, fireballs, 3D water simulations, and other effects. “Joe [Letteri] set down the hard line,” Williams says. “He told us not to plan on cheating anything.” At one point during postproduction, the studio was generating 110gb of data an hour.

“Jim Cameron’s expectations are extremely high, and he demands a lot,” Rosenbaum says. “The scope of CG movies is getting so large and the time constraints too tight, that people tend to compromise, but Jim doesn’t compromise. He insists on a high standard. When I worked on The Abyss, it took us six months to create 90 seconds with the pseudopod. We went into it with the same question we had on this film: How the hell will we do this? And we had the same mind-set: We’ll put our heads together and figure it out. He’s always one to push a VFX company. And he certainly did it on this one.”

http://www.cgw.com/Publications/CGW/2009/Volume-32-Issue-12-Dec-2009-/CG-In-Another-World.aspx

The director of Terminator and Titanic explains how movies will be transformed by motion-tracking and 3D technology

Three-time Academy Award-winning director James Cameron is a pioneer in the field of motion capture. In the mid-’90s he used the nascent technology to create the massive crowd scenes and stunts in his blockbuster Titanic. These days he’s still at the cutting edge of the technology, but he prefers to call motion capture “performance capture” because, as he points out, “actors don’t do motion, they do emotion.”

Cameron is in the midst of his latest film project, Avatar, which is his most technologically innovative film to date. The futuristic movie about an ex-Marine will be released in 2009 simultaneously with a massive, multiplayer, video game based on the film.

Business Week couldn’t catch up to Cameron for a sit-down interview, since he’s busy creating Avatar, but reporter Aili Mc Connon was able to engage the director, via e-mail, in a discussion of how motion-capture technology has spurred innovation in cinema and made filmmaking more cost-effective. The following are excerpts from their virtual conversation:

What has motion capture meant to the film industry and to your work?

Performance capture (Perfcap) in recent years has enabled such stunning [computer generated] characters as Gollum (in Lord of the Rings parts 2 and 3), “King Kong,” and Davy Jones (in Pirates of the Caribbean) to be brought to life. The technology is critical to the realization of my dream project, Avatar.

In fact, Avatar wasn’t possible when it was first written 11 years ago, and only through pushing the technology to new levels over the past year and a half have we reached the point where the film is finally possible to make.

What innovations have you developed for Avatar?

We have greatly enhanced the size of the performance-capture stage, which we call The Volume, to six times the size previously used. And we have incorporated a real-time virtual camera, which allows me to direct [computer-generated] scenes as I would live-action scenes. I can see my actors performing as their characters, in real-time, and I can move my camera to adjust to their performances.

In addition, we have pioneered facial performance capture, in conjunction with our visual effects partner, Weta Digital. This technique eliminates hours in the makeup chair, and various other discomforts, for the actors. Previously, actors needed to have hundreds of tiny spherical markers glued to their faces, and they couldn’t touch their own faces throughout the shooting day as a result. With the new system, a lightweight head-rig can be donned minutes before shooting.

We have had great success, and other filmmakers such as Steven Spielberg and Peter Jackson have worked on our virtual stage doing tests for their upcoming films, and given high praise to the system.

Does the rig cover the whole head, including your face? Does it capture fine facial movements?

The rig is a small skull cap, made from a cast of the actor’s head, so that it fits comfortably while being tight enough to avoid shifting. It acts as a base for a strut which resembles a concert microphone (visualize Madonna in concert), except instead of a mike in front of the face, it has a tiny camera. The key to it is the software, which interprets the movement of the actor’s face, pupils, and eyelid responses as the image flows in from the video feed of the head-rig camera.

In what directions do you see the technology going in the short term?

Improvements to the software and higher computation speeds and storage densities will enable us to have more realistic environments and more refined facial emotions and hand movements. Hand movement, for example, is still at a crude state.

On Avatar, we’re working on-stage at a reality level equal to an ’80s video game. At the end of the day, after a year and a half of post production, the images seen by audiences will be 100% photo-real, i.e. indistinguishable from photography. But for our day-to-day shooting, the image can be improved a lot.

Another area which needs improvement is the lighting. We need to improve its ability to handle cinematic lighting, the casting of shadows and so on. All of this can be improved as Moore’s Law raises the speed of processing and as upgrades to the software become available.

In addition, we’re developing ways for [computer-generated] characters to interact with actors who are being photographed on real, live-action set. We will have real-time stereo (three-dimensional stereoscopic, or 3D) composites of characters, which will be viewed by me in the eyepiece of the camera while I’m shooting a live-action scene. This will be revolutionary. We’re not quite there yet, but we hope to have that by August, in time for our live-action shoot in Wellington, New Zealand.

Long term, what do you expect?

I expect that more filmmakers will embrace the technique and apply it to different types of scenarios. For the creation of fantasy and science-fiction characters, Perfcap will largely replace makeup and prosthetics.

Actors need not feel threatened by this change in technology. It doesn’t replace acting, in fact it’s designed to empower the acting and directing process, as opposed to the traditional [computer-generated] animation process, which uses only the actor’s voice, and in which a committee of animators perform the character, operate the camera, and do the lighting.

I believe it will make fantasy filmmaking much more user-friendly for filmmakers, actors, and studios, and ultimately bring down costs. It’s just now possible to create photo-real human [computer-generated] characters, but it isn’t cost effective.

Many other fields, from medicine to automotive design, now use similar motion-capture systems (though on a smaller scale). Do you ever run across or dream up non-entertainment applications yourself?

I’m bore-sighted on the cinematic process. While one can generally imagine all the industrial and science applications, I’m not interested in developing them. However I can visualize a number of uses for the technique in advanced forms of entertainment, at theme parks and so on.

What role will 3D play in the future of film?

Here’s what can happen, although it’s too early to say if it will: 3D can become ubiquitous as digital cinema replaces film. As digital cinema rolls out, stereo follows—and in some cases leads the charge, as we have seen recently with the digital 3D releases of Chicken Little and Monster House forcing the installation of hundreds of new digital projectors.

There will eventually be major titles available from all studios at some screens in almost all multiplex cinemas worldwide. I would say the ho

rizon for this is five years. 3D can become a fully accepted way in which audiences view movies. It will become another consumer choice, like premium or regular gas. The premium experience of 3D will be the preferred viewing experience for action, animated, fantasy, and science-fiction films.

3D’s broad acceptance at theaters will generate enough content that consumer-electronics manufacturers will make home players and monitors available. The technology exists now, but is not readily available as off-the-shelf products. 3D display will become a must for video and computer games.

In 20 years, stereo media may become the preferred method for displaying all information, including news and other broadcast media. The density of information one can place on a small screen becomes much higher if it’s stacked in three dimensions.

Is there something beyond 3D in film? Could we ever see in cinema the same kind of physical participation we’re starting to see in video-game consoles like Nintendo’s

Wii?

Imagine a movie in which the viewer is swept along by a narrative, following the action from place to place, but without the intervention of a camera. You can choose which character to watch in a scene, as if you’re an invisible witness standing there while a real event plays out. This is still years away, at a level of realism people would consider cinematic, but certainly not decades away.

I can imagine the dense fantasy worlds I like to create for movies having an equal or greater life in a world of interactive play, authored by others, in a partnership. Of course, add massive multiplayer capability to this, and people will never leave their homes.

“I was calmly laying out the next decade of my life when The Hobbit appeared,” he laughs. “I was preparing all these things and all of a sudden The Hobbit shows up and takes over my life.”

Make no mistake: The Hobbit is his precious. Del Toro knows more than anyone that this diptych could – should – define his career.

And so the director has been busy building a world that not only honours JRR Tolkien’s book and Peter Jackson’s Lord Of The Rings trilogy but will emerge assuredly, triumphantly, his own.

Our very own cuddly character, Jamie Graham, snuggled up to Del Toro at his Wellington base of operations, and talked exclusively about the biggest films of next decade.

The Hobbit has taken much longer to design than your other movies…

It took almost a year. Which for me is very, very long because normally I take about a third of that time to design movies like Hellboy. And if you actually take into account we have three or four times the number of artists… [chuckles].

We produced hundreds, literally hundreds, of drawings; dozens and dozens of maquettes; dozens of material tests. It’s epic. And we are still going to be designing into production…

How did it work with the writing of the script? Presumably you’ve had as much input as Peter Jackson, Fran Walsh and Philippa Boyens?

Many, many months ago we sat down to discuss the structure with 3in by 5in cards and we laid out the two movies.

We were meeting on a daily basis at 9am and we would go at it for hours, into the afternoon. Then in the afternoon I would go to check on design.

Then at one point we split into two teams: I did one pass at things and they did a pass at things; it’s pretty much the way I’m used to co-writing.

But I must say what was great and what made a big difference was the amount of great ideas that I felt were generated in a day – it was staggering.

We could have written three or four versions of The Hobbit [laughs].

You mentioned the structure. Will the book make up the first movie, with the second movie plucked from the appendices and maybe even your imagination? Or will parts of the book be saved for the second movie?

We are respecting the structure established by Professor Tolkien because the order of the adventures in The Hobbit is well known to generations and generations of kids. You don’t want to be moving stuff like that.

But we will be integrating Gandalf’s comings and goings because he does disappear in the book quite often.

So, as opposed to the book, we see where he goes and what happens to him

You and Peter are both visionary filmmakers who will fight for those visions. What happens when you clash?

So far we haven’t come to such a crossroads. We argue and we win at different stages. But I think Peter has been, so far, the perfect producer.

Two filmmakers have produced me in my life, both named Peter. One was Pedro Almodóvar and one is Peter Jackson.

Both times my experience has been that they are perfect producers because they understand the producer is not a producer/director.

A producer is a producer. If there’s an emergency, if everything goes wrong, then the producer can – and should – have a strong opinion.

But while everything is going well, on time, on budget and is creatively solid, there’s no need for that.

Presumably working with Peter is not that much different to working with Mike Mignola on the Hellboy movies?

You nailed it. I’d say Mike is as opinionated as if he was another director because essentially he directs on the page. And Mignola, like Pedro and Peter, knows the process – they all know that at some point you’re going to be alone with the beast [laughs].

You’re going to be the guy and you can only trust your own instincts.

You’re not going to be making a phone call from a remote location to ask a question; you’re going to have to make a decision yourself.

So how arduous has it been commuting between LA and Wellington? You’re now in New Zealand full-time, yes?

Yes. I go to LA very seldom now. It is, however, an incredibly easy commute for me. I’m used to it. I’m used to London-LA and in the same way I’m used to Wellington-LA. I blob out on the plane [laughs] and I have 13 hours all to myself, so it’s a privilege.

I write, or prepare emails, or read, so it’s a really great working day.

And the great advantage between LA and Wellington is that you are essentially in your time zone. You lose a day but you go to sleep in your night in LA and you wake up the next morning in Wellington.

Do you find time to sneak in the odd movie on the plane?

I do! But I try to watch television mostly because it doesn’t need you to have a big screen!

You love creating your creatures and obviously The Hobbit offers some great opportunities. There’s the dragon Smaug, the spiders of Mirkwood, the Wargs, Beorn the bear-man…

The way I phrased it to Weta, I said we would keep the DNA in the same gene pool as the Rings trilogy, but that we would generate a different type of character. For example, in the trilogy most of the creatures are brutish or inarticulate.

In The Hobbit, the creatures speak: Smaug has beautiful lines of dialogue; the Great Goblin has beautiful lines of dialogue; many creatures do. So we had to design them with a different approach because you are not just designing things that are scary.

I also wanted some of the monsters in The Hobbit to be majestic.

I wanted the Wargs to have a certain beauty so that you don’t have a massively clear definition: what is beautiful is good and what is ugly is not. Some of the monsters are absolutely gorgeous.

Smaug won’t be like the dragons in Reign Of Fire, say. Was it a big challenge to communicate his character?

I think one of the designs I’m the proudest of is Smaug. Obviously he took the longest.

It’s actually still active: we’re finishing his colour palette and a little bit of the texture. But the bulk of the design took about a year, solid. It’s because of the unique features of the dragon.

Early in production I came up with a very strong idea that would separate Smaug from every other dragon ever made. The problem was implementing that idea. But I think we’ve nailed it.

What was the idea?

I cannot tell you what it was because it would be a massive spoiler! But I’m 100 per cent happy with Smaug. If there is such as thing as 110 per cent, then I’m there!

What about the spiders? How faithful are they to Shelob from Return Of The King?

Well, they are the progeny of Shelob, but Shelob was quite a promiscuous girl [laughs]. She mated with many partners. And insects and spiders are incredibly adaptable creatures. There will be spiders… [Laughs]

That sounds like a Paul Thomas Anderson s

equel: There Will Be Spiders! But they are visually quite striking and in a different way to Shelob.

I wish I could tell you more but I would be spoiling it again. They are very different. They are more creatures of the shadow, more creatures of the deep forest. They are not earth nesting. They are nesting in the canopies so physically they have adapted to that environment.

Will the sequences involving Smaug and the spiders be genuinely scary?

I think so. I hope so. At least that’s the way we’re approaching it. Every good children’s movie, be it early Miyazaki or Disney, always has a thrilling scene or two. When I read The Hobbit as a kid… Well, you have the moments like when Beorn has the heads of goblins on spikes outside his house [laughs].

Tolkien made no bones about that. There is no way to have a dragon attack a town that’s not scary. It’s the same for the spiders: there is no way of making giant spiders cocooning people so it would be gentle!

Have you been studying real spiders? There are some big ones in New Zealand!

We have been. We have a couple of the guys in the design team who are obsessed with spiders.

They actually do their own little documentaries and features and they go out and capture spiders and they shoot their mouthparts and this and that with macro-lenses.

The main problem with the spider designs is how do you translate the weight into a design so nimble or so long-legged, because a spider has long legs. With Shelob, she was quite low to the ground so she moved like a tank. Our spiders have to feel massive but be very nimble.

Are you OK with the real spiders?

No. I adore insects, and I’m completely fascinated by spiders… But I am completely and absolutely horrified by them, too [laughs].

It’s something that Peter and I share!

How about the scale of The Hobbit? You’ve done big action sequences in Mimic, the Hellboy movies and Blade II, but you’ve never tackled anything like the climactic Battle of Five Armies…

No – and I think that I’m really quite eager to go and do that. But at the same time there were so many battles in the trilogy. So one of the first things is how do we make the battles or the action in The Hobbit feel different from that?

Because it was fresh when the trilogy came out, to see those enormous valleys or fortresses being invaded by warriors.

But then after the trilogy you had Troy, Narnia, everything. It has become quite common seeing two massive CG armies attacking each other.

So we came up with a good solution, I think. It will make the battles stand out.

Is it going to be more intimate?

I wish I could spoil it! All I can say is that we have an incredibly good team of people who know we are not making another Rings. We are not trying to make a quadrilogy, or a pentilogy. We’re tying to make two films that flow with those but that stand on their own completely.

We want to avoid stuff that is not part of the DNA, that is not part of the lexicon, but we also don’t want people to feel “We’ve seen this”.

Except where that familiarity is comforting, like Hobbiton or Rivendell – then you want to feel like you’re coming back home to a movie that you love and cherish.

Will you be using the same palette as the trilogy, dark and fertile?

I think The Hobbit is a bit more colourful. And a bit more operatic. And whimsical. One of the things the book marks very strongly is the seasons, so we’re using that as the basis of our thought.

Presumably it will also be a bit more magical? Have a stronger fairytale vibe?

It is in many ways just what you enjoy in the book. You enjoy an almost chamber piece, like when the stone trolls talk about cooking the dwarves.

It’s such a small piece but at the same time it’s magical and it’s almost a comedy, that you have these enormous creatures talking about cooking these dwarves!

It wouldn’t be a Guillermo del Toro movie unless it possessed a poetic quality, surely?

There is a lot of magic in the film. Peter has the eye of a strong historian, in the sense that the trilogy is incredibly accurate to a world that was created. He’s like an archaeologist who’s digging

something that existed. I think that The Hobbit has a little bit more poetic licence.

It has… How can I say it? It has a little bit more flamboyance.

The Hobbit begins shooting in late spring 2010 and will open in 2011.

http://www.totalfilm.com/features/guillermo-del-toro-on-making-the-hobbit

http://www.massivesoftware.com

Wired: What tips would you offer an actor doing motion-capture for the first time?

Serkis: For digital roles, the actor is manipulating their character like a puppet. It’s really useful to have time on a monitor to work with the CG model — to play around with your puppet before the actual shoot. It’s like having a third eye on yourself. Actors have to learn to demand that time.

Wired: Did your role in Heavenly Sword expand over time, as in The Lord of the Rings?

Serkis: I got more and more involved in the character development and the writing. When I took the actors down to New Zealand to rehearse, we sat in a circle and performed the whole game, from beginning to end, as a play for each other. By treating it as theater, we could see how all the characters were inter related, figure out where scenes weren’t working, and feel the whole arc.

Wired: Were you already an avid gamer when you took the gig?

Serkis: No, but Heavenly Sword got me much more into it. I’m not bothered by hack-and-slash games, but what I really enjoy is being taken on a journey to other realities. I have a strong desire to create games from Shakespeare — play as Romeo, play as Juliet. Macbeth is an amazing story. Maybe I should be keeping these ideas to myself (laughing). One thing that’s going to change in the next few years is that scripts for games are going to come more from the dramatic arena. They’ll be more like film scripts. You can’t just come up with an idea for a game and stick the drama on top. It all has to be one driving thrust.

Wired: While you were growing up, you spent a lot of time in the Middle East.

Serkis: My father’s Iraqi — he’s a doctor, retired now. My mum moved me and my older sisters to London when I was a year old, but my father still had a practice in Iraq. I stayed in Baghdad every summer until I was 14. My dad’s sister is still there, but many of my relatives have managed to get out. People forget that there are still people there who are not radicalized in any particular direction, trying to live normal lives in a very difficult situation.

Wired: What experiences in your early acting career prepared you to do motion-capture for Gollum and Kong?

Serkis: My first job when I got my equity card was acting in 14 plays back-to-back. Playing that many roles, you look for ways of differentiating the characters physically, which goes hand in hand with understanding them psychologically. In 1992, I played a homeless kid called Dogboy in a play at the Royal Court Theatre called Hush. When his dog is killed, he allows the creature’s spirit to possess him, and he breaks into this middle-class household to avenge his spirit. I was naked for the entire performance. There was a lot of Dogboy in Gollum.

Wired: Were you surprised at how much input you ended up having on The Lord of the Rings?

Serkis: It was very much an organic process. I got a call from my agent who said, “They’re looking for someone to do a voice for a completely digital character. It’s going to be three weeks’ work.” But then I met Peter Jackson, and he said, “No, we’re looking for someone to be Gollum on set, because we want real chemistry with the other actors.” I learned that the only way that I could generate Gollum’s voice was by fully inhabiting the character.

My first day, I was climbing down the side of a 6,000-foot volcano in a Lycra suit, and the crew was like, “We thought Gollum was going to be animated. Who the hell is this guy who looks like he just walked out of a fetish shop?” That was terrifying. But as everything came together — the motion-capture, rotoscoping, animation, voice and breath work — the process became very exciting. Nothing like it had ever really been done before.

Wired: Have years of cyber-acting changed your approach to stage acting?

Serkis: It’s made me more still. My natural bent is to have an overabundance of energy, and motion-capture essentializes your every breath, your every move. Seeing yourself through that mask, you realize how far you can pull back and make the performance even more powerful.

Wired: What’s on your wish list as a digital actor?

Serkis: The environment you’re working in for performance-capture is very clinical. There’s no stimulation from sets or costumes; you’re working in a black box with lots of lights around you. I want to be able to shoot a scene in costume instead of a Lycra suit. We need motion-capture studios that let directors use lighting, back projection and other forms of stimulation to help the actors feel immersed in the world of the film.

Wired: What projects are you working on now?

Serkis: I’m in the early stages of a film called Freezing Time about Eadweard Muybridge, the Victorian photographer who was really the forefather of cinema. Digital animators still treat his images like the Bible. He was a very obsessed man. He tried to have a relationship with his wife, but it wasn’t fully consummated, so she ended up having an affair with this dashing guy called Harry Larkins. Muybridge shot him dead in a fit of jealousy but was acquitted because the murder was considered a crime of passion.

I’m also working on a movie called Inkheart with Brendan Fraser and Helen Mirren based on a book by Cornelia Funke, who is like the German J. K. Rowling. It’s about an antiquarian bookbinder who has the ability to “read” characters out of books. I play a very dark character called Capricorn who is accidentally read out of a book and doesn’t want to go back in. Then the bookbinder’s wife falls in. That’s coming out next year.

Wired: You got dissed by the Academy because Gollum was considered a collaboration with the animators at Weta Digital. Will a CG character ever win an Oscar?

Serkis: For The Elephant Man, a whole team of prosthetics artists worked on John Hurt’s character to help him create that performance. Whether or not the Academy can learn to see ones and naughts as a digital form of prosthetics — that is the question.

http://www.wired.com/entertainment/hollywood/magazine/15-10/pl_serkis

In this article, I’m getting right at the meat of what technologies are, getting us closer to that goal, and what this really means for the future of not only motion capture filmmaking, which Steve Perlman of Mova (www.mova.com), refers to as volumetric cinematography, but also the impact it will have on live-action films and mixed media films, like James Cameron’s upcoming Avatar. Mocap is finally coming of age, but the future will be even more exciting.

In the simplest terms, the future of mocap will be a seamlessly integrated motion capture experience where the limitations imposed by current technologies are overcome and the mocap process blends into the background. This will allow us to capture full body and facial motion data, as well as capture the movement and texture of real clothing, skin, props, and environments, while permitting realtime compositing or superimposing with live-action elements.

The idea is to create a mocap environment with all of the advantages of traditional filmmaking — with few of the drawbacks and incredible flexibility in post — not available in traditional cinematography. This is an ambitious though daunting order, but it may not be far off.

NEW WAY OF VIEWING MOCAP

Traditionally, mocap has been viewed as the capture of body and facial motion, translated into animation that drives the motion of 3D characters. While this is still a very big part of mocap, in reality, it goes far beyond character animation. Motion capture often starts with capturing body and facial data, but in many cases, virtual cameras also simultaneously use motion capture to create a perspective and live preview of the actions being performed by the actors. In post production, additional camera motion tracking may be used to set up camera placement and movements. Since each element can be manipulated independently, replacing dialogue, or doing reshoots may only require mocap for one character since the existing mocap for all other characters likely does not need to be changed.

MIXED MEDIA MOCAP

While directors like Robert Zemeckis have embraced mocap for every aspect of their filmmaking environment, others, like James Cameron have used it in conjunction with traditional live-action environments and actors. This creates a whole new set of challenges, but also opens up a lot of new opportunities for interactively combining live-action characters with mocap characters. Glenn Derry, virtual production supervisor on James Cameron’s Avatar (December 2009) explains that although the movie is being filmed with the stereoscopic Pace Cameron Fusion camera system, and works with live-action characters, 75 percent of the movie is virtual. In order to achieve their filming style, they have had to develop a number of tools that allow them to mix live-action actors with mocap virtual characters, as well as combine traditional sets with virtual environments. For example, if they were filming a scene in which an avatar is interacting with an actor, they might be working on location, but using a motion capture set-up that combines an active optical marker tracking system, with inertial sensors, and camera tracking.

Cameron called on Atlanta- and LA-based Giant Studios (www.giantstudios.com)  for its proprietary, realtime mocap technology. The Avatar stage is in Playa Vista, CA.
Derry describes their shooting workflow as very director-centric, meaning they use a digital camera with a virtual eyepiece, which shows a live view of the combined superimposition of the live-action elements along with the virtual characters, which are being rendered and superimposed in realtime, as well as composited greenscreen elements, whose movements are matched to the camera movements based on realtime camera tracking. In other words, the director sees all of the elements of a shot combined in realtime, for a live preview. In this way, the director chooses his shots during filmmaking, which has the advantage of the actor acting for the camera. This can be helpful since an actor adjusts the size of his movements based on the framing of the camera. It is also necessary to use this approach when combining live-action characters since the view of them is fixed by the position of the camera while filming the live-action sequences.

In many ways, they are already achieving many of the goals of future mocap in that they have broken free from filming in a mocap volume. They have done this by combining multiple technologies, and have also succeeded in realtime preview of the superimposed images, making many aspects of the motion capture, just part of the background process.
what’s new?

There are several new technologies that promise greater accuracy and resolution. Others offer better realtime preview and data management, while others offer the flexibility to do them anywhere. Some even approach the capture from a completely different perspective, and promise to capture the motion and texture of cloth and skin, while having scalable resolution options.

HMC

When I visited ImageMovers Digital (www.imagemoversdigital.com) and the set of Robert Zemeckis’s A Christmas Carol this past April, I got to see all of their new toys, including their new facial motion capture system, the HMC or Helmet Mounted Camera set-up, which they developed with Vicon. The HMC is a system with four small cameras attached to small booms, placed low on the face, just below the view of normal visual perception. These are mounted to a lightweight skullcap helmet. This captures multiple perspective video of black dots marked on the actors’ faces, which allows for individual facial motion tracking data to drive the facial motion of an individual character. This can significantly speed up the post process by requiring fewer cameras to capture body motion and keeps the facial motion data for each character separate. This allows for much more reasonable amounts of data to track, solve and retarget. This also helps with realtime preview since selectively, only the body motion has to be solved. It also is useful for early versions of scene renders since facial animation can be postponed until camera positioning and movement have been determined.

The HMC is not dependent on a capture volume, since the cameras go with the actor  and don’t require any sort of fixed stage. This could ultimately be used to drive the facial movement of a character interacting on a normal set or location, assuming facial replacement is the only requirement.

The HMC isn’t the only kid on the block, though. Glenn Derry has been using a single camera facial capture rig they call Headrigs, in Avatar, which predates the HMC. “It also employs a skullcap helmet, but uses a single camera mounted from the side like a headset microphone,” he says.

While lacking the multipositional perspective for tracking, as well as the resolution of a four-camera system over a one camera system, it makes up for it with realtime facial motion solving and playback, which Derry explains is what is most important for their application, and provides them with the results they need in their streamlined, realtime workflow.

RESTROSPECTIVE VERTEX TRACKING

Mova’s Steve Perlman has created a unique approach, using some new and existing technologies, which has some real potential to do things that haven’t really been done before… at least not like this. The Contour Reality Capture system takes a bit of a step backwards, in that, for now, anyway, it places the actor in a very limited environment, similar to the earliest days of facial mocap where the actor had a very limited range of movements. From there, however, it takes a couple of big steps forward, and assuming they can increase the volume to a reasonable size, it could have a big impact on the mocap industry.

The system uses some tried and true concepts, but uses them in a unique way. Several high resolution black and white cameras set in sync with UV lights flickering 120 times a second to capture the phosphorescent glow of UV makeup applied to skin, and UV dye applied to clothing. These multiple perspective cameras will act as the motion capture sources, but the points that are being tracked haven’t yet been determined.

Using a technique referred to as Retrospective Vertex Tracking, Mova’s software uses pattern tracking of the random pattern made by the sponge application of the phosphorescent make-up on the skin. Since the points that are tracked aren’t determined until post, different resolution solves can be made based on the need for more or less resolution, and greater resolution can be applied on areas of the face that need more tracked points. Because of the way it chooses tracked points, it has potential of having much higher resolution solves that traditional tracking marker systems.

At the same time, a UV image is being captured for motion capture. Several regular color cameras are set directly out of sync with the UV lights, but in sync with 5600K fluorescent bulbs. These effectively capture the texture map images which are projected back onto the 3D mesh, basically translating the video into a 3D video of the person’s face, adding  some of the subtle details like minor wrinkles and semi transparent skin.
These are things that don’t register in mocap, but can be seen on video. The result is a 3D mocap, which looks very real. This also works with fabrics and uses no visible markers.

THE FUTURE OF MOCAP VOLUMES

While the ability to capture motion data outside of a mocap volume is critical for some applications, the mocap volumes still have big advantages in terms of accuracy and the ability to view multiple subjects from any angle. This is what still makes them the tool of choice for non-mixed media apps. With this in mind, those who use passive-optical-based mocap volumes are constantly improving and upping the ante with their stages. An example of this is Vicon’s House of Moves (www.moves.com), which has just completed construction on their new mocap volume.

Their new stage can be configured in 30-x-50-feet for full body capture only, or 30-x-30-feet for full body, plus facial and finger capture. This stage differs from their old stage in that it is all white, with 270 near infrared cameras, which is more comfortable to work in since it is easier on the eyes. More importantly, however, was the design of the volume as a traditional soundstage in that it is sound proofed and designed to capture production grade audio, eliminating the need for most ADR.

The cameras are configured in two separate systems so that finger and facial capture is separate from the body motion capture. This allows Autodesk Motion Builder to be used to drive the animation of a virtual character in realtime. They have streamlined their workflow to timecode sync video, animation and audio, and have integrated custom virtual camera rigs with realtime preview to allow the director a tactile, interactive perspective. Hand-held virtual cameras add to the functionality. In addition, Vicon has integrated greenscreens in their volumes for intermixing live-action characters with motion capture characters.

Imagemovers Digital’s mocap stage, which was built for A Christmas Carol, also uses near infrared lighting, but had a separate truss system built for wirework so the mocap cameras wouldn’t be effected by truss movement. Instead of upping the camera count, they relied on 100 cameras, using the HMCs for facial motion capture. Lightstorm Entertainment in conjunction with Giant Studios also relies heavily on mocap volumes, using a 70-x-36-foot volume for performance capture for Avatar, and use a single camera facial capture system they call Headrigs.

JUST THE CAMERA PLEASE

One of the areas of motion capture, which is seeing a rapid increase in demand, has nothing to do with character animation and everything to do with camera placement and movement. This is camera tracking, which has actually been around for a while now, originally used for greenscreen apps so virtual cameras would match the live-action camera movements for compositing live-action characters into 3D environments. Now, however, some of the same systems, like Intersense’s IS900 studio camera tracking system, are being used by directors like Zemeckis to set up camera angles and movements in post.

When it comes time in the post process to bring in the camera, Zemeckis breaks out his IS900, which he has hooked up to three machines. Then, one at a time, he uses the virtual camera to frame his shots and create camera movement. He moves from one computer to another so he can optimize his time, allowing each 3D artist to tweak the movements while he sets up the next shot. In this way, he knows that the shot he is working with is a perfect take, so he can focus all of his attention on getting the right angle, framing and movement. It is a critical part of his post workflow.

Traditional passive optical mocap stages all use specialized virtual camera rigs that act as a physical representation of a virtual camera with some camera and navigation controls. These feed a viewfinder playing back the virtual camera’s view. Some directors prefer to set up their shots as they record the motion capture, whereas others prefer to focus on the capture and deal with the camera set-up details in post. Each has its advantages, and represents different directing and workflow styles and preferences.

INERTIAL TRACKING

Inertial tracking is another mocap technique that is seeing greater implementation. Inertial trackers generally work by having accelerometers and gyroscopes in them that detect motion in any direction along any axis. These provide all sorts of useful information. These sensors can be used independently, with inertial trackers placed at joints and other logical locations on a motion capture suit, like those used by Moven. Or they can provide supplemental information when used with optical systems to provide directional information to help with solving tracking points when they have been blocked from view. Intersense uses a combination of acoustic tracking and inertial trackers in their IS900 camera tracking system.

The advantage inertial sensors have over most other motion capture systems is that they can provide 3D movement data for each inertial sensor without having to be observed by outside sources, like cameras in optical systems or acoustic sensors in acoustic systems. They can be used under clothing, and in most shooting environments. The limitation is: they work well for body capture, but have no solution for facial capture and aren’t practical for finger capture. Used in conjunction with other technologies, however, they have a lot of potential for use in mocap production.

NEXT-GEN WORKFLOW

In many ways, the techniques used by James Cameron on Avatar, now being employed by Weta Digital for Steven Spielberg and Peter Jackson’s latest mocap film, Tintin, is a model of what can be expected to come. On Avatar, the approach is to shoot everything in the same way as you would a normal movie. What they have done, however, is mix in the mocap technologies and feed the characters and environments created in 3D, which are driven by the camera tracker and performance mocap, back into the viewfinder in a live environment in realtime.

The director knows what he is shooting and how it will all be composed when completed. In fact, as they are filming the movie, Cameron is cutting the film, so what is sent off to Weta has already been cut to the frame. This director-centric approach gives the director a very tangible view of what he is capturing. What is lost is the freedom that you have in a completely mocap production. In essence, each of these tools had been designed to best suit both the production style of the directors and, more importantly, work within the environment, restrictions, and requirements demanded.

BIGGER, BETTER, FASTER

As it stands, in the world of mocap there is always a tug of war between increased accuracy and realtime performance. The greater the resolution and number of cameras, the more points can be captured with greater the accuracy of the points and the reduced incidents of occlusion. Unfortunately, this comes at the cost of realtime preview. The flexibility of shooting anywhere, especially outside of a soundstage or volume restricts the normal flexibility of 3D motion capture in that it often solves only the mocap from certain perspectives, which is fine when you are mixing with a traditional camera since its perspective is determined at the time of recording.

As processing gets faster and the tools, which combine multiple technologies, are developed, we can expect higher resolution while still providing realtime playback. There will also be a larger number of tools, and a larger pool of talent. The systems will be more flexible, quicker to set up, provide more accurate data, and provide more integrated workflows.

WHAT IT ALL MEANS

As technical as motion capture is, ultimately you will hear the same thing from virtually all camps, “It’s all about story.” The purpose of these technologies is to allow us to create characters and effects that we could not do before, and do it more realistically and efficiently than ever before. They allow us to create and interact with virtual worlds and blend the lines between reality and fantasy. The tools are becoming more flexible, more powerful and are changing the way a lot of films are being made.

Heath Firestone is a producer and director for Firestone Studios LLC (www.firestonestudios.com), which specializes in mixed media 3D compositing and camera tracking.

http://www.postmagazine.com/ME2/dirmod.asp?sid=&nm=&type=Publishing&mod=Publications::Article&mid=&tier=4&id=EBB3ED7515214734AA0BAA618B57011E

Heath Firestone

Motion capture, or mocap as it is often referred to, is one of the great new frontiers in the world of movie making, and although it has had its resistance in the film community, it is becoming a crucial tool in complex digital effects. And, as it develops, it has become a completely different medium in which to capture entire feature films.
Robert Zemeckis has produced three films entirely using mocap: Polar Express, Beowulf and A Christmas Carol. Peter Jackson, James Cameron, Steven Spielberg and George Lucas have also used the technology in their productions.

Although some early attempts at motion capture yielded sometimes plastic-looking facial movements and dead eyes, the technology is continually improving and opening the doors to a future way of making films, which Steve Perlman of San Francisco’s Mova (www.mova.com), refers to as “Volumetric Cinematography.”

DEFINING MOTION CAPTURE

To describe mocap simply, it is a conversion of live-action movement into 3D data, which is used to drive 3D animated characters. How this is accomplished is not as simple to describe and spans the spectrum from single perspective pattern tracking interpolation, mechanical armature exoskeletons and radio frequency triangulation to the more commonly used multicamera optical marker-based triangulation systems. There are even a couple of system out there, which use gyro-based inertial sensors and sonic sensing using time of flight triangulation for positional information.

Mocap in the film industry, however, more commonly uses one of the passive optical-based systems, like those made by Vicon and Motion Analysis. This is starting to expand into more advanced systems, which use more than just tracked motion data, including capturing textures and cloth surfaces. Motion capture is also being used for realtime capture of virtual handheld camera movements, which Scott Gagain, executive producer at LA’s House of Moves (www.moves.com), says, gives the shots an organic feel, and can also be an invaluable tool for previsualization. (I will cover this topic in more detail in a later issue of Post, when I write about the future of motion capture.)

A BRIEF HISTORY

Motion capture has been around in various forms, for over a century if you count Eadweard Muybridge, who in 1878 filmed a series of 12 photographs showing a full stride of a horse to demonstrate that all four of the horse’s hooves leave the ground at one time, mid-stride at the point where all four legs are tucked under the body. While this isn’t 3D motion capture as we know it, it was the beginning of analysis of motion using high-speed photography, which is the basis for motion capture. Since then, motion has been captured and analyzed mostly for scientific and medical purposes, but more recently has found its way into film production.

Since films first started using 3D models, it has been the goal to create realistic motion for those models and speed up the workflow so the motion doesn’t have to be hand animated. The solution is to capture the movements of an actor in three-dimensional space and apply that to the 3D models… in other words, mocap.

MOCAP IN THE MOVIES

Motion capture has been used to drive computer animated characters in movies, such as Lord of the Rings, The Mummy and King Kong, or as a way of enhancing and matching partial animations on live-action characters, which can also be seen in The Mummy. It has also been used in movies that have relied on the technology for the entire production, starting with Sinbad: Beyond the Veil of Mists, Final Fantasy, Polar Express, Beowulf and the upcoming A Christmas Carol. Programs like Massive also draw from libraries of mocap data for the animations that power their autonomous AI creatures in movies like The Lord of the Rings trilogy.

HOW DOES IT WORK?

Since all of the films that have relied heavily on motion capture have used passive optical motion capture set-ups like the ViconMX, which has been used on all of Robert Zemeckis’s mocap films to date, I’m using Vicon’s workflow to describe how it works, although there are other solutions out there.

Motion capture takes place on a stage, called a volume, which has a number of monochromatic, high-speed, high-resolution cameras mounted all around the stage. These cameras are optimized for picking up a certain range of light, usually red or near infrared, and have a couple of hundred high-intensity LED lights packed around the camera lens. These LED lights surround the lens in order to act as the light source that illuminates the retroreflective markers affixed to the actor, which are the points that are being tracked in 3D. The actors generally wear tight-fitting body suits that have retroreflective markers (round balls covered in retroreflective material) velcroed to them, and/or retroreflective dots glued to an actor’s face. Retroreflective material reflects back to its source very brightly, because it is coated with millions of microscopic glass beads, which — through refraction and reflection — bounce light off of the back “lens” of the bead, back toward the source of the light. This causes the material to ‘light up’ brightly, even with low intensity lights. But for this to work the light has to be coming from the same angle as the lens you want it reflected back to.

Robin Pengelly, senior VP/manager of LA’s Vicon Entertainment (www.vicon.com), explains that the cameras have improved, where they no longer need to capture in front of black backgrounds because their custom cameras use filtering to eliminate non IR lightwaves, so the camera sees a high contrast image where the retroreflective markers appear as white dots in a black environment.

(Near infrared LEDs are usually used for close-up, facial motion capture, primarily because they are not hard on the eyes, being barely visible, whereas, red LEDs are brighter from the camera’s perspective, and can be used for a lot longer distance motion captures, as on a full stage.)

The motion capture cameras are placed around the volume at different perspectives, then a calibration tool is placed and moved around within the volume. This gives the software the information it needs to determine where each camera is in the volume. When multiple cameras are used at different perspectives, and the software knows their location through calibration, the software has all of the information it needs to cross reference which dots it sees traveling in which direction from different camera angles and triangulate its three dimensional location in the volume. This process is similar in all volumetric mocap  techniques in that multiple perspectives are needed to determine three dimension positional information through triangulation.

More cameras are used when more detail is needed, or multiple subjects are being captured, in order to have better coverage, and less likelihood of occlusion of markers. Occlusion is when a marker is blocked from the view of any of the cameras, making it impossible to track the marker.

After the motion data has been captured, it goes through a stage called Cleaning, which is where the software filters and resamples the marker data. It handles occlusion either through software, which uses techniques including estimating position based on trajectory and velocity in the case of live capture, or in post it can look at the position it is in before it is occluded and after it reappears and create a path between the two. Or it may be manipulated manually, where an artist may modify the motion curve to make a more realistic movement or correct data that the software was unable to calculate correctly.

Solving is the stage where all the information from the cameras is run through the software and converted into a 3D representation of the marker points. When several hundred or thousand markers have been tracked, the resulting points look like a mass of points, which is referred to as a marker cloud.

These tracked points then need to be attached to the corresponding positions on a 3D skeleton, which is in turn, attached to a 3D character. This process is called Retargeting. In some cases, this will have been pre-rigged so a live version can be displayed using a tool such as Autodesk’s Motion Builder, which is software that converts the animated skeletal information into a live animation of a 3D character driven by the motion capture.

EVOLUTION

The world of motion capture has evolved a great deal over the past decade. It has gone from working on a dark, empty stage with dozens of bright red ring lights surrounding the stage — where the actor had to imagine the environment with a restricted number of simultaneous motion capture subjects — to a point where mocap-friendly sets are built with interactive props and facial motion data is being captured in the same take.

Set pieces are still usually just metal frames, but props are often actual objects that have been altered to be mocap friendly — made non-glossy. Ben Guthrie, motion capture shoot lead at San Rafael, CA’s ImageMovers Digital (www.imagemoversdigital. com), which is a Robert Zemeckis company, explains that although the intention of using mesh props was to be marker friendly, the mesh tended to break up the view of the reflective markers, making it difficult to track the center of the marker accurately. So for smaller props, they have shifted back to using more realistic, but marker-friendly props. The addition of interactive props, built stages and 3D models to use as references helps in the envisioning process, as actors no longer have to imagine everything, though they do have to look past their mocap suits and concentrate on performance. Fortunately actors seem to adapt quickly and seem to focus on playing off of the other actors and communicating with the director without the hindrance of normal production distractions.

THE ACTOR’S CHALLENGE

Actors in any film production have several challenges in connecting with their character and being in the moment of the scene, despite the hot lights and dozens of people scurrying behind the scenes fixing makeup, adjusting lights, capturing audio, etc. On a mocap stage, or volume, they don’t have to deal with as many of these distractions but, at least for now, they generally don’t wear normal costumes and are instead on an open stage being bombarded with the light of several thousand near infrared lights, surrounded by hundreds of cameras, often while up to 170 retroreflective markers are affixed to their faces. Fortunately, the cameras and near infrared lights blend into the rigging, and LED lights only emit a dim red light, which can barely be seen. The stages have also evolved from dark Duvateen covered walls with bright red rings every few feet to bright white stages with nearly invisible light rings.

While they have the freedom of knowing they don’t have to exactly hit a specific mark, since this can easily be adjusted in post, they can concentrate on performance instead. Debbie Denise, VFX executive producer for Sony Pictures Imageworks (www.imageworks.com), explains, “Theater actors generally have an easier time adapting than method actors, as they are often less reliant on costume and environment to connect with their character.” Theater actors tend to be more used to working on a limited stage, playing to large audiences, while keeping the intimacy of the scene.

“They’ve already made that leap,” she explains. A lot of actors who haven’t yet worked with mocap are at first apprehensive, but, although it might seem unnatural to act in spandex clothing, with semitransparent props, opposite actors who are also wearing spandex, once they get used to it they often feel a freedom of being able to focus on the craft and don’t have to reshoot an otherwise perfect take because of a change that needs to be made to lighting, makeup or costume. In the world of mocap, everything else is in post. So if you got the performance you wanted, then it is captured and all other tweaks will be created in post.

In the case of movies, which are entirely shot in mocap, as is the direction Robert Zemeckis has chosen, I was curious how the shoot schedules compared to traditional filmmaking. Denise explains, “Shoot schedules are one half to one third the length of traditional shoot schedules, often with only 25 to 30 days of shooting scheduled, and these aren’t long days.”

ZEMECKIS’S JOURNEY

It all began on a little film called Polar Express. Robert Zemeckis, no stranger to technology and pushing the envelope, delved into his first fully 3D motion capture film. It wasn’t the first, Final Fantasy and Sinbad preceded it, but it was considered groundbreaking for other reasons. Imageworks’ Denise explains that when they did the initial tests for Polar Express, mocap was done in two passes, one for full body capture and the other for facial capture. Facial capture was very limited in movement. Not satisfied with that solution, Zemeckis wanted to capture facial and body motion simultaneously and worked with Vicon to get a system going that would have that capability. On Polar Express, they had three different volumes. One was 10-foot-wide-by-10-foot-deep and 16-foot-tall. At first, they were only able to handle one or two actors at a time, but eventually were able to bump that up to three. They also had a 20-foot-by-20-foot stunt stage and a 30-foot-by-60-foot stage for doing crowd scenes.

Although Zemeckis didn’t direct Monster House, he was executive producer, and it built on what was learned from Polar Express. In Monster House, they were able to get away with less facial motion capture data because of the stylized nature of the film. They shot Monster House on a 20-foot-by-20-foot volume with 240 cameras.
When Beowulf came around, Sony Pictures Imageworks tested other RF-based tracking options, hoping to be able to use traditional costumes because they wouldn’t have to rely entirely on optical markers, but the technology wasn’t available in time for shooting. Instead, they bumped up to 240 cameras on a 25-foot-by-25-foot stage, capturing as many as 20 actors with up to 250 markers each, including facial motion markers on all 20 actors. That’s a ton of data.

On Zemeckis’s most recent film (A Christmas Carol, still in production), now under the ImageMovers banner and Disney, new technologies were used, which allowed them to break away from the difficulties associated with capturing that volume of markers and gave them the capability to capture each face individually. It also allowed them to scale back their camera count down to 100 on their 30-foot-by-60-foot stage. This new HMC (Helmet Mounted Camera) system uses four tiny cameras mounted to booms, which attach to a simple, lightweight helmet and use ink dots on the face instead of markers. Since the cameras are mounted low on the face, almost out of view of the normal range of perception, they are easy to ignore unless the actor is trying to touch his face or drink, etc. This along with other new technologies starting to be used in mocap feature film production will be discussed in more detail in a follow up article.

ON THE HORIZON

“The reality of motion capture is that there is no magic bullet, each system has limitations,” says Steve Boyd, co-producer at ImageMovers Digital, who has been involved with Polar Express, Beowulf and A Christmas Carol. “The Holy Grail would be if we were able to capture not only the body movements and facial animation, but also be able to use regular costumes with no markers on the face, and be able to do this outside, not restricted to a soundstage.” Ideally, we will be to be able to capture all of this information and have this part of the filmmaking process be transparent. To accomplish this would probably require a combination of technologies.” These technologies have come a long way, and each film pushes the envelope and builds on the experiences of the last film.

Although Zemeckis took some pretty harsh arrows from several critics on Polar Express, he weathered the criticism and has embraced and advanced the technology to a point where the quality has become good enough that other prominent directors have felt comfortable taking the plunge. This includes James Cameron, who has reportedly made the leap to fully digital, motion captured characters after seeing the work being done and determining that it was finally at a point where he could use it on Avatar. He is also using a system with a camera attached to his actors, which captures facial motion data similar to Vicon and ImageMover’s HMC.

The one caveat about using very realistic 3D faces, is that you have to steer clear of the “uncanny valley” no man’s land where people respond very negatively to a face that is realistic enough that people almost believe it is human but because of certain missing subtleties of movement make parts of the character look somehow not quite right. This is sometimes described as having dead eyes, or plastic-looking facial movement. Since the goal in filmmaking is to keep people involved in the story, distractions — even subconscious ones — need to be avoided. This means, either getting it right or backing off to nonrealistic characters, which was the approach taken on Monster House.

One of the most exciting new developments in this area, which is being used on films (though which ones, have not yet been announced), is Mova’s Contour Reality Capture, which Mova’s Steve Perlman describes as a capture system designed for a new form of filmmaking he refers to as Volumetric Cinematography. It is also a multicamera optical capture system, but rather than capturing points it captures the random patterns in sponged-on phosphorescent makeup applied to the actor’s face, from multiple angles, and is synced to be captured only when illuminated by flickering UV lights, which are timed to be offset from the flickering of white Kino Flo lights. Offset from this are multiple color cameras capturing the white light exposure in order to get texture images, which will be mapped back onto the 3D model for advanced realism.

What is truly different about this system is that it uses what Perlman refers to as Retrospective Vertex Tracking, which means that the tracked points are determined in post, and the number of tracked points can be scaled up or down — depending on need. Because it doesn’t use fixed dots, it allows for far greater potential point resolution than existing solutions, but for now stage size is still limited in scope.

WHERE IT IS NOW?

Mocap has established a strong foothold in the industry and is gaining popularity and acceptance. It creates possibilities in filmmaking, which did not exist before, and with the use of virtual camera tracking and more advanced capture techniques, will change the way that many films are made.

In a follow-up article, I will talk about the process of how this is really changing filmmaking for directors like Zemeckis and the future of motion capture. It’s an exciting time for mocap, with advances being made daily.

Heath Firestone, a producer/director based in Denver, has a strong background in advanced 3D digital effects and compositing. Reach him at Heath@FirestoneStudios.com.

http://www.postmagazine.com/ME2/dirmod.asp?sid=63930E9D02644EC9A127AF67602DA756&nm=E-Newsletters&type=Publishing&mod=Publications%3A%3AArticle&mid=8F3A7027421841978F18BE895F87F791&tier=4&id=C715B81DD6674D62BD666D304D2E8D0B