the whole "i had friends who smoked weed and went on to do X" argument really doesn't prove anything other than a lot of people smoke weed nowadays and statistically many of them are likely to try "harder" drugs, who often times are more open about their drug use which creates a incredibly skewed perception of how "stoners" are perceived.
on the other hand all the incredibly successful, intelligent people (who i know a lot of), tend to keep their marijuana use more secretive for the exact reason you stated in your post; the fear of peoples reaction to their use and possibility of negative consequences (ex: loss of scholarships, jobs, legal issues etc)..
also kinda off topic but LSD is not a Hard drug, in fact it is literally one of the most harmless substances known to man. in fact the inventor of lsd (a swiss scientist called dr albert hoffman and regular user of the substance) died at the ripe old age of a 102 in 2008...the more ya know....