I didn't produce the visor myself. I was lucky enough to get it along with the helmet (I can't remember who it was that actually sculpted the helmet) but I received it from Tony of stony props awhile back when they started forming up. He put a good amount of effort into making adjustments to the visor to make it fit tighter/better, but it helped only so much, as there were multiple gaps between it and the actual helmet.
What I ended up doing was attaching the visor to the helmet (this was a pain as some parts were persistent in not staying down!) and using steel putty, filling the gaps all around the main/lower portions of the helmet. It took along time and a lot of careful work, as I had to sand it down and essentially "re-sculpt" the lower "jaw" details. In the end, the visor is now pretty flush to the actual helmet and appears much nicer IMO.
To answer your question; yes, the visor was indeed the worst part
