Richard Frazer with a great post on creating nice and believable soft edges on a composite in Nuke using Color Smear. Solor Smear will smear out the edge colour of your A plate to create better soft edges and Works by blurring and un-premultiplying your image.
Check out the post for Creating a Color Smear in Nuke for Better Soft Edges here.
If you have to roto something with a soft edge and composite it over a new background, a common problem is that the colour from the original background will bleed through. You see this as colour fringing around edges where there is heavy blurring, and the new / old background have very different colour values. The blurring in the plate means that your subject has been effectively premultiplied over the original background, and no matter how accurate your roto matte is it will not allow you to accurately unpremultiply it. If your subject were against a background with a consistent colour (such as a greenscreen) then there are ways to despill the colour, but if not then you need a different approach. This technique helps to re-introduce the original colour of your foreground object into these soft areas, whilst maintaining the desired softness of the matte.
Richard notes that: “I have seen different artists do similar techniques in a number of ways, but this is my take on it (the EdgeExtend gizmo is a simpler version – this one combines the edge erosion aspect and produces softer results). This technique is not application specific, but I am demonstrating it here in Nuke (although I doubt you could do this in After Effects as you need to be able to manually control pre-multiplication).”
