The hardest part when starting up is definitely getting all of your tools and equipment in place and working properly to build. The more time you spend making sure things are perfect here, the better end product you will get.
As far as the actual building process goes, profiling cores is tough. With the old planer I have it just takes a long time and pretty much all of that time I'm scared shitless that my planer is going to tearout/chew up the core and make it unusable (I have had that happen). Aside from the layup, it's really the most critical part of the process because you're trying to get the flex dialed and a bad move here can definitely force you back to square one.
The other difficult thing can be finding the time. I wish I started sooner when I didn't have to deal with life as much back in highschool. Now, working M-F 8-5 can be a bit of a pain in the ass to build good momentum and keep it up, but it's still pretty doable. It might be even tougher if you're a student with a job or have longer working hours. Just have to make sure you capitalize on spare time when you have it and everything will go smoothly.
Everything else is pretty easy, it just takes some time and experience to get good at, but like skiing you will definitely see a progression if you do a few pairs.
Financially? It's expensive to start up, no doubt about it. I was lucky to have a lot of tools at my workplace that I needed (and more) to get into building so I was able to save some cash there. Aside from that though I bought and payed for everything else, and after the smoke cleared I was in around the three grand mark to get the first pair out.
That was a year ago, and now I'm laughing. I've spent about $900 on materials for this season which will do about 10 pairs of skis. So the return is pretty damn awesome especially when compared to the average market price of a ski.
If you have any questions shoot me (or some of the other NSers who are into this crazy awesomeness) a PM. Good luck with it!