MayaStaion with another great tip, this time on a new feature of Maya 2011 that can help with memory related issues. The “Allocate on Heap” option you will find under the mental ray options tab > performance and it will allow the shading engine to store temporary data on the heap (dynamically allocated memory).
From the Documentation : “When raytracing through transparent materials, the shading engine stores temporary data on the rendering thread’s memory stack. Occasionally, a rendering thread may run out of space on the stack, causing Maya to stop unexpectedly due to a stack overflow. To workaround this, select this option so that the shading engine stores temporary data on the heap instead of on the memory stack. Selecting this option causes mental ray for Maya to render slower, but it may prevent Maya from stopping unexpectedly due to a stack overflow when using a large number of render passes in a scene with many layers of transparent objects.”
