From the Documentation : “When raytracing through transparent materials, the shading engine stores temporary data on the rendering thread’s memory stack. Occasionally, a rendering thread may run out of space on the stack, causing Maya to stop unexpectedly due to a stack overflow. To workaround this, select this option so that the shading engine stores temporary data on the heap instead of on the memory stack. Selecting this option causes mental ray for Maya to render slower, but it may prevent Maya from stopping unexpectedly due to a stack overflow when using a large number of render passes in a scene with many layers of transparent objects.”
