Good observation, IamS.
I happened to save the PS file for this demo so I was able to go back and look at what I did. Even so, I'm not sure exactly what I did to introduce the jaggies.
The path from Illustrator was as smooth as silk (at any magnification). The pixel-filled area from Illustrator was almost as nice - almost no trace of stair-stepping.
I thought I had used exactly the pixel-filled Illustrator output as the layer mask as shown in the screen grab (below) of my layer stack, but when I now look closely at it, the layer mask was not exactly the same as what came out of AI. I'm not sure how I messed it up, but the version I used as the layer mask definitely had the jaggies and was the cause of the problem.
To check this was indeed the problem, I re-created that adjustment layer, and this time was uber-careful to use the correct layer mask. The jaggies are about as small as you can get without the edge being too soft (see attached).
Maybe I mindlessly thresholded the output from AI or did something else stupid yesterday.
BTW, you used the phrase, "clean it up". FWIW, I've tried lots of after-the-fact methods to clean up aliasing, stair-stepping and other such artifacts, and never have been particularly impressed by any of them. About the best of the lot is PRP's Anti-aliasing plugin (
http://powerretouche.com/Antialias_plugin_tutorial.html ), but like most things in life, fixing the cause is always better than attempting to fix the result ;-) .
Again, thanks for pointing out the problem.
Cheers,
Tom