After actress Nancy Marchand died last year following a long battle with cancer, the producers of the popular HBO drama The Sopranos wanted her character, Livia Soprano, to reach closure on air. The result demonstrated that a television character could be brought back to "life," but raised some important moral questions about responsibly using technology.
Visual effects supervisor for episode two, freelancer Chris Scollard, contacted Rhinoceros Visual Effects & Design (RVED), New York, about the ambitious possibility of manipulating Marchand’s past performances so that Livia Soprano could have a last conversation with her son, Tony. RVED jumped at the opportunity.
Scollard presented RVED with an Avid cut which consisted of James Gandolfini—who plays Tony Soprano—performing his lines with a body double, and 18 shots featuring Marchand from the first two seasons of the show. Now in its third season, the David Chase-created series centers on an Italian-American mobster family in New Jersey.
"Our job was to composite her performance onto the body double’s head," explained Rick Wagonheim, RVED executive producer/partner. RVED had three weeks to complete the project, which aired as a double episode on March 4. It was also previewed at New York’s Radio City Music Hall, where it was screened on HD projectors.
The scene, which is four minutes long, shows Marchand " talking" to Tony Soprano, but the two are never actually seen face-to-face.
The RVED team involved with the job included David Binstock, founder/CEO RVED’s parent company, Multi-Video Group, New York; Wagonheim; Michael Miller, executive producer/partner; Yfat Neev, senior producer; Niklaus Schlumpf, Inferno artist; Craig Gordon, technical director; and John Binninger, senior colorist.
None of the original 18 shots of Marchand really matched. This is largely because they were selected for the lines she was saying, rather than for how she looked.
Discrepancies included different hairstyles (sometimes her hair was parted on the left, sometimes on the right) and different lighting. In some shots, Marchand’s hands were in front of her face as she spoke, but the body double’s hands were also visible. Additionally, the backgrounds didn’t always match; for example, in some shots Marchand was sitting in front of a wall, while in others, in front of a window.
"We were pretty amazed when we saw the Avid cut with a rough composite. You could see how different it was, but you didn’t see just how different it was until you started looking at the negative," said Miller.
Actors have been "resurrected" before—most notably in the 1994 feature film directed by Alex Proyas, The Crow, after Brandon Lee died during filming; and in Ridley Scott’s Gladiator, after Oliver Reed’s death. But Wagonheim told SHOOT that, as far as he knows, this is the first time something like this has been done for a television series. "It was also created in broad daylight because the scene was shot in the middle of the afternoon, and it was close-ups so we couldn’t hide anything in a wide shot," he added.
But don’t hit the panic button yet. Digitally created performances are unlikely to replace actual performances anytime soon. "I don’t think there is going to be more demand for this now," predicted Wagonheim. "I think maybe a special circumstance may call for this, like an actor’s death in the middle of shooting a movie. [But] the reality is, you can’t replace the spontaneity of a live performance. On a technical level, yes, there are instances where 3-D people have been created, and you are getting to the point where you can do 3-D facial expressions and manipulations. But if you can ever do something live, … that’s the approach you always take first."
But Miller thinks demonstrating that a character can be brought back to "life" digitally will open up a lot of avenues. "Every time some ground-breaking visual effects get done, they open up another set of doors to creative options that start treeing out," he stated.
THE PROCESS
The scene had up to 15 layers of compositing in some of its shots. As the Inferno artist on the job, Schlumpf was charged with replacing the body double’s head with Marchand’s. Once the footage had been color-corrected and transferred to HD through a digital disk recorder, it was loaded into the Inferno. Because the footage came from many different episodes, RVED needed to go back to the original negative. But some of the frames were missing where the negative had been cut; therefore the adjacent frames needed to be morphed.
Once this was sorted out, the job demanded the meticulous rotoscoping of the footage of Marchand’s head. "Because a lot of the movement was in her head and she was supposed to fit onto somebody else’s body—who was moving differently or not moving at all—first we had to stabilize the footage so her head stayed at the same place," Schlumpf explained.
"She also put her hand over the face," he continued. "Removing it required a lot of morphing techniques. Sometimes she was saying words when she was covering her mouth, so we had to re-morph her lips to have her look like she was saying what she was supposed to. After that, we had to track her head onto her body double and finally make it look like part of the scene with the color correction of the different parts, matching the lighting and changing her hair. For example, in one shot you could see her collar through her hair, and in the next scene her hair was longer and covering her collar."
"On top of that, Schlumpf added, "the body double put her hand over her face, so we had to rotoscope her hand out and create shadows in Inferno."
Apparently Chase was skeptical up until the last minute; he had the footage enlarged and watched it frame by frame, in HD. In fact, there was never any guarantee that the manipulated scenes were going to wind up in the final episode. "It wasn’t until executives from The Sopranos showed up on the last night and gave it their blessing that we knew it was going to be used," said Wagonheim, who maintains that the response from the public was more positive than the reaction of the press—some of which was brutal. One columnist described it as "macabre exploitation."
"Questions came up about the moral and ethical issues involved, and where technology is heading. Our point of view all along was that it was an incredible tribute to an incredibly written role, an incredible actress—and that’s all it was: nothing more and nothing less. We were surprised by it, we were proud of it and we were flattered by the praise we were getting," he concluded.
