If you are lighter and have a low calf muscle it may be fine. depends on the boot and your foot, size and ability level/ style. Go to a boot fitter or at least learn what to look for in a properly fitting boot.
These people don't know what they're talking about. What really happens if you use a women's boot is you start producing more estrogen, and slowly your body starts developing more like a women's body.
wore some women's scarpa touring boots for 3 seasons. one of the best boots I have ever owned and they were super cheap because not too many women wear a size 11.