I fell face-first onto a knuckle of a big table with mine on - the visor did nothing to help the impact. The screw ripped out and the visor crumpled I guess. I came away with a black eye. I keep the visor on for just looks and minor protection from tree branches/etc.
and besides, the visor isn't in a fixed position you can move it up or down about an inch so either it will just break off or the visor will push up and nothing will happen. but it certainly won't twist your neck.