Rendering Transparent Textures in an Ambient Occlusion Pass With Maya Mental Ray
Here is how to create an ambient occlusion pass with mental ray that will respect the transparency of materials
Ever set up a render including an ambient occlusion pass, and then notice that your objects that are using transparency to create the shape are just rendered as the object and not the shader? Take for example, Yun Khang Lai’s image of a simple scene that uses flat or curved planes in 3D space for leaves and foliage, using a texture that “cuts out” the shape of the leaf from the 2D plane. this is quite a common practice in 3D as it makes so much sense for things like leaves etc.
It’s tricky to get textures with alpha channels to show up in AO and Z Depth passes and among the most frustrating issues in some 3d apps. In this quick video tip, I go through exactly how to do this in MayaYun Khang Lai - TalesofNalstone.com
Mental Ray Ambient Occlusion Pass Issues
The problem arrises when you want to render out different passes for the scene, and find that the ambient occlusion and depth passes render out as 3D planes, both ignoring the textures with transparency. Of course this will look wrong in the composite as the AO render and the beauty render are no longer the same.
Use Maya’s Render Layers
Here Yun Khang Lai provides a simple and straight-forwad solution for creating ambient occlusion and depth map passes that are forced to respect the shaders that utilize and alpha channel. Its not magic, and requires a lithe bit of work, taking the render passes in question back to using Maya’s render layers.
You will often hear me complain about the integration of mental ray in Autodesk Maya, (or lack thereof). Truth be told, the addition of the mental ray render passes was a huge leap forward in workflow. This made it possible to grab plenty of AOV’s pretty easily in a render, and made Maya less reliant on the render layer ideology.
Render layers in Maya were pretty flexible, but really a bit tedious to set up, but as Yun Khang Lai notes, it can be done fairly quickly. Relegating the ambient occlusion and depth passes to Maya render layers will allow you to add the shaders as you go – so not really an automated solution, but it does work.