I think that the key to understanding why the observation of pixelation when zooming in on a smart object is perceived as an unusual effect / problem can be found in these statements:
"...the quality was awful when zooming in a little...." - Post #1 by the OP
"... The reduction looks great at 100%. ... As soon as you zoom in it starts getting ugly..." - Post #9 by Sam
First, I think it's safe to say that anyone who has even a beginner's level of knowledge of PS expects ALL non-vector images to look pixelated if zoomed in past 100%.
The "problem" occurs because of a mis-perception of what smart objects are supposed to do and how they work. Smart objects only store the initial state of the object before any transform or other operation is applied to it, and they store the final state, after the transform. HOWEVER, THEY DO NOT KEEP RECOMPUTING THE FINAL STATE AS YOU CHANGE THE VIEWING MAGNIFICATION RATIO, as I suspect the OP had hoped.
For example, in the case being discussed in this thread, the smart object stores the full rez version of Sam's mountain, and it also stores the version after it had been shrunk to 20% of it's original size. When the user asks to view the result at some different magnification ratio, it takes the "after" version and magnifies or shrinks it in exactly the same way that it would display any pixel containing layer. If you zoom in substantially past 100%, it gets pixelated.
The benefit of smart objects is that if, say, after looking at the 20% result, you realize this is too small and you really should have only shrunk the object to 60% of its original size, not 20%, had you been using an ordinary, non-smart layer, you would have little choice other than to expand the 20% version by 3x in linear dimensions. This is guaranteed to produce a horrible result. In contrast, if you had been using a smart layer, when you change the parameters of the transform to produce a 60% version, it pulls up the "before anything happened", 100% version and computes the new result from it, not from the 20% version, an obviously much better approach. This is a tremendous benefit, especially if you wind up tweaking the size multiple times to see what gives you the best "fit".
HTH, and hopefully I correctly understood the effect causing the concern.
Tom