How an Average VFX Pipeline Works
These are the steps that take Hollywood films from the green screen to the big screen.
Top image: The Jungle Book via Walt Disney Pictures
Visual effects are so prevalent in modern cinema that it can be easy to overlook the extensive process involved in every computer-enhanced shot. While behind-the-scenes breakdown videos are becoming increasingly common, few of them reveal what a VFX pipeline is actually comprised of. Here’s a look into the standard creative steps of creating on-screen magic.
Research & Development
The first step in any VFX pipeline is usually research and development. During this period, the technical approach to the film’s effects is decided, focusing on software preference and technique. Most Hollywood blockbusters will require unique programs or plugins more sophisticated than mass-produced VFX software, and it is during the R&D phase that programmers, artists, mathematicians, and even scientists develop the tools necessary to bring the on-paper concepts to life.
For instance, the stunning black hole, Gargantua, as seen in Christopher Nolan’s Interstellar, required its own proprietary physics simulator to be built from the ground up, as well as extensive scientific input, namely from Kip Thorne, the Feynman Professor of Theoretical Physics at California Institute of Technology.
Pre-visualization is essentially the process of converting a storyboard and script into a 3D animated, low-quality rough draft of each VFX shot. While this does not always accurately reflect the succeeding final result, it allows the director to get an idea of what shots will look like and therefore prepare better for principal photography.
Pre-vis technology has become extremely sophisticated as of late, even overlapping into the production stage by providing live, digital environments that can be referenced during the shooting process. As you can see, this technology is a significant part of what made Jon Favreau’s recent adaptation of The Jungle Book possible.
One of the most extensive and crucial aspects of CGI is 3D Modeling. This process actually occurs throughout all three stages of production, with iterations of varying detail being created for varying purposes. For instance, pre-visualization only requires low-detail representations of the final product.
Image via MovieViral
Depending on the importance of a given digital prop, it could be comprised of anywhere from hundreds to millions of polygons. Built according to concept art, or in the case of real objects, according to reference photos and 3D scans, 3D models are the foundation of any digital subjects.
While 3D models make up the bulk of an environment, many backgrounds are still created using matte painting. Matte paintings have been commonly used to create background landscapes since long before the rise of digital effects, but they remain as important as ever.
Image via SlashFilm
While they are often created digitally now, as opposed to with traditional paint, matte paintings are still the favored method of producing immersive backdrops.
Principal and Reference Photography
During the actual shooting process, certain members of the VFX team will usually be on set. While they do not have much input into principal photography itself, they will take numerous photos of everything, from the environment to individual, seemingly-unimportant props. These photos are used to create textures for 3D models and to reference the scene’s real attributes such as lighting, object size, and more.
The more detail involved in reference photography, the more realistic the resulting VFX should appear. If shot on film, the VFX house is usually provided with their own copy of the film scan after shooting, with which to create a more detailed, High Dynamic Range version for reference and tracking.
Scene preparation covers a number of processes, but they all serve the same purpose: prepare the provided footage for elements to be inserted in. Among these are motion tracking, rotoscoping and keying, and color correction.
Motion tracking analyzes footage with a moving camera and/or moving object, and tracks the 3D motion of the footage to replicate the camera’s movement and place objects into the scene convincingly. Without motion tracking, VFX could not be inserted into shots with camera movement, nor could they be seemingly attached to moving objects.
Image via AgenZasBrothers
When analyzing camera movement, this is technically called match-moving, while motion tracking is usually regarding individual object movement.
Rotoscoping and color keying are the techniques used to cut out elements from a live-action shot. Color keying removes a specified color from a scene, as seen with green screens and blue screens.
Rotoscoping is a more tedious alternative to keying, and involves cutting out an object or person frame-by-frame.
Color correction is the process of adjusting the lighting and color profile of each shot to meet a unified, consistent look. This is not to be confused with color grading, which establishes the creative look of a movie after all of the visual effects are completed.
If shot on film, the film scans being used will also undergo a process called plate preparation (or “dust busting”) in which artifacts like scratches and patches are removed from the film scan.
Rigging is what enables 3D models (such as characters and vehicles) to move. A 3D model of, say, an animal, is initially static, but once rigged, its limbs and body can be adjusted and given motion.
Image via FlickeringMyth
Realistic movement, especially of existing creatures and machines, is highly dependent on proper rigging.
Animators add the movement to rigged models, and are responsible for much of the physical realism of CGI. Animators will often test rigs and attempt to “break” them, sending them back to the rigging department for refinement until perfection.
A professional animator must have a thorough understanding of the visual subtleties of physics that they can accurately portray in their work.
Explosions, water, and smoke are all simulation-based effects that come to fruition here.
Polishing effects (ranging from lens flares to fake cigarette smoke) are added over the existing scene, at which point the final product starts to become vaguely visible.
Texturing is the process of adding a surface color and texture to the 3D models, making them recognizable and now near completion. This is where metal objects are given their shine, and humans are given their skin. The level of detail given to an object’s texture is dependent on how close the camera gets to said object, and is therefore usually proportional to the level of detail given to the 3D model itself.
One of the most important factors in realistic CGI is the fidelity of its lighting. Good lighting will accurately imitate that of its setting, and due to the sheer difficulty of the process, poor lighting is often one of the first culprits in revealing an element as fake.
Daytime lighting is especially arduous to replicate accurately, which is why many CGI-reliant scenes are set at night.
At last, the final stage of the pipeline has arrived! Composition is the step often showcased in VFX reels, where everything comes together to assemble the completed product.
The live footage, the scene preparation data, the matte paintings, and the various VFX renders are given to a compositor, who then combines and blends them together to create a single, seamless image. The movie will still be color graded after this point, but that is usually out of the hands of the visual effects department.
There are undoubtedly many less significant steps of standard VFX production that went unmentioned here, but hopefully this at least gives you an idea of the immense amount of work put into every single CGI moment you’ve experienced.
What stage of the VFX pipeline do you find the most interesting? Let us know in the comments below.