The smoother the buck, the smoother the visor. Imperfections like ridges, bumps, etc. will come through in the visor and cause a distortion in vision. Due to the concave shape of the visor its a little bit like looking through a fish bowl. So ideally the smoother you can make it, the better your vision through the visor will be.
In my experience a step down approach to sanding would be 120, 220, 400, 600(wet). If you want a SUPER clean visor a final wet sand with 1000. YMMV.