Improve Render Quality with Layer Composition in After Effects
Having complete control over your renders has never been so easy. Just follow this in-depth, step-by-step process.
On many occasions, you’ve probably heard of layer composition as a way to operate in post-production in order to make your renders even better.
Imagine a situation where you want to tweak some reflections, but the render has already been made and it’s practically impossible to isolate your reflections, unless you start your render from stratch.
Consequently, the only way to make changes without affecting the other parts of the scene is to consider layers.
Layers and Render Passes
When you approach rendering, you have to find a way to isolate different elements in separated layers. That doesn’t necessarily mean the background from the foreground, but mainly the type of light rays affecting your scene.
For instance, a renderer knows what diffuse rays, as well as reflective rays are, but, if we don’t manage render passes, the render engine would include everything in the same layer.
Therefore, a render pass is an image containing specific aspects of the scene. The same render pass is being imported as a layer in After Effects.
Here, we have an example of a rendered scene, which I created for this article. I decided to manage reflections, subsurface scattering effects, and refraction for the glasses.
Building the final render with all the layers included, known as a Beauty pass, is our primary goal.
Compositors have to take all the passes and create the final image, as we see below:
Most Common Render Passes
There are many ways to compose your final image by isolating specific light rays and elements. Almost all of the software allows for rendering specific layers. For this purpose, I’ll be using Maya with Arnold, but choose the renderer of your liking.
Here’s the list of the most common render passes that we’ll take into account:
- Diffuse direct considers the diffuse color of a surface due to direct lighting. The pass stores the direct diffuse rays from the light sources in the scene.
- Diffuse indirect shows the indirect diffuse rays connected with secondary or tertiary bounces. Colors are soft—this pass can be tweaked in After Effects to make colors more vivid or to emphasize the effect on the global illumination.
- Specular direct indicates the specular reflections connected with the direct lighting.
- Specular indirect renders the indirect specular reflections. As you can see, the table reflects the dishes and the vase with a bit of roughness.
- Transmission direct and indirect are responsible for the rays passing through surfaces like glass, liquids, and so on. I decided to introduce a few examples of refractive surfaces. This pass isolates the transmission rays, as we can observe in the following image.
- SSS direct and indirect: The subsurface scattering is typical of materials having some translucency like milk, wax, leaves, and so on. In our render, the candle is the only element where light enters, diffuses internally, and scatters back. The SSS pass captures that.
Other Render Passes
- Z–Depth is generally used to add some depth of field in post-processing. It presents itself as a grayscale image, which stores the distance of the scene objects from the camera. We’ll see later how to set up this pass in After Effects.
- Object ID: The purpose of this pass is to mask some objects in the scene and apply local effects like color grading, exposure, and so on. The renderer assigns a uniform color to an object so that After Effects can easily isolate it.
- Ambient Occlusion adds soft shadows in occluded areas and that gives more realism to the scene.
- Mask: We don’t have anything in this pass, but it simply represents a mask whose purpose is to separate the 3D content from the background. Imagine a character composited on a real background.
- Shadow: Sometimes, you might want to separate the shadows from the rest of the composition and increase or decrease the strength independently. In our case, we don’t want to have a separate pass including shadows, so we’ll ignore this pass for the rest of the article.
From your 3D software, you can save these render passes as a single EXR file. This file contains all your layers packed in.
Once completed, you can use it in your compositing software. But, how can we read the single layers from a single file in After Effects?
EXtractoR to the Rescue
Here’s the core of the article. Let’s take the previous render passes in After Effects and create the layers for the final composition. You can then tweak the single layers and have more control over your final render.
The plugin EXtractoR is what we need. It takes an EXR image file from your 3D software and splits the render passes in AE layers. It’s included in the latest versions of After Effects, but it’s a free plugin anyway.
In this article, I’m using AE 2020.
Let’s import the EXR file containing all the aforementioned render passes and drag it into the timeline. Look for the EXtractoR effect and apply it to the layer.
What you have is the Beauty pass. We see reflections, refractions, translucency, and so on.
The cool thing about EXtractoR is the possibilty to select a specific render pass from its Layers menu. A dropdown opens and you can pick one of the render passes from your 3D software.
We want to rebuild the Beauty pass by extracting layers. Here’s how we do it:
- Duplicate the main layer several times and give the layers a significant name. Delete the Beauty pass layer at the end.
- Each layer must have its own EXtractoR effect applied, with a specific item (render pass) selected from the Layers menu.
- We have 9 layers associated with 9 render passes.
At the moment, ignore the Z-depth and ID passes which will be configured later. We also assigned a blending mode for every layer—Normal for the diffuse_direct layer, Multiply for the ambient occlusion, and Add for the rest.
Here’s the timeline with all the exploded layers:
We’ve just rebuilt the Beauty pass, but with the possibility now to tweak every single layer. Let’s do it!
Coloring the Indirect Diffuse Bounces
Let’s boost the indirect diffuse layer a bit by applying the Channel Mixer and Hue/Saturation effects. We want to have a warmer tint in the indirect diffuse light bounces by mixing the red with a yellowish color, and by adjusting the saturation.
The effect is subtle and mainly evident on the ceramic vase, the wall, and the candy jar.
Tweaking the Specular Reflections
I decided to apply some changes in the specular direct layer by adding the Exposure effect with a value of 1.29. You can observe more reflections on the candy jar and the glass.
For the specular indirect layer, I decided to leave it as is, without any change.
The first image shows more reflections than the second one.
Setting up the Transmission Color for the Carafe
One of the cool things you can do with the glass is to change the transmission color. Some wavelengths are absorbed while others pass through the refractive surface. That gives a tint to the transparent objects.
Furthermore, I decided to tint the carafe on the left by leaving the other glasses untouched. I achieved that with the use of a mask.
In the first image, you can see a subtle orange color appearing on the carafe. I applied a Photo Filter effect with the color set to yellow/orange and a density of 43%.
Working on the Candles
For the candles, we want to change the color to a pale pink.
In AE, I took advantage of the Curve effect to increase the contrast a bit. Nevertheless, the change in color is obtained by using the Hue/Saturation effect. A decrease in saturation and a couple of tweaks to the tint and we’re done!
Ambient Occlusion and Object ID
The ambient occlusion pass is already being included in the previous renders. I simply considered it to create some soft contact shadows on the table. In AE, you can play with the layer opacity, but don’t leave it at 100% because the effect might be a bit strong.
In relation to the Object ID pass, I decided to isolate one object (the candy jar) and change its color.
The setting in AE is quite simple:
- Consider the ID layer we created before and apply the Linear Color Key effect to it. By manually selecting the corresponding key color, we isolate the jar.
- Create a new adjustment layer just below the ID layer.
- Consider the column Track Matte of the adjustment layer and check the option Alpha Inverted Matte ID, which masks to the jar.
- Finally, add the Hue/Saturation effect to the adjustment layer.
At the end, we have a new colored jar! Remember, you can apply the same procedure for any object in your Object ID layer . . . very powerful pass!!
Z-depth and Final Touches
As previously stated, the Z-depth pass is mostly used to create some depth of field in the composition.
The black parts of the image tell After Effects to keep the area on focus, while the white parts mean out of focus. There’s a gradual passage from black to white, though.
The Camera Lens Blur effect is what we need! It allows you to specify a blur map (in our case, the Z-depth pass) and adjust some parameters to refine the depth of field.
Follow these steps:
- Be sure to have the previous Z-depth layer in the timeline with visibility set to hidden.
- Create a new adjustment layer and call it Z-depth Adj. Layer.
- Apply the Camera Lens Blur effect to it with the following settings . . .
Finally, you should have something like this:
- Create a last Adjustment Layer at the top of the timeline, with Levels and Photo Filter effects applied, in order to add more contrast and make the whole environment a bit warmer.
Let’s compare our final render to our initial Beauty pass produced by the EXtractoR plugin. The result is much better!
In this extensive tutorial, we explored one way to create a nice composition in After Effects. We used the EXtractoR plugin to separate the different render passes, coming from your 3D software.
At the end, we improved the quality of our render by applying effects to the single layers.
In relation to the render coming from your 3D software, I suggest you start using an ACES workflow to drastically improve the overall quality of your works.
I hope you found this tutorial helpful and I invite you to follow me at my Artstation page!
We’ve got a few more tips and tricks up our sleeve. Take a look at these beauties: