Dirt biking can make for awesome adventures too, depending where you live.  I'm from a big logging town in BC and the logging roads through the mountains are unreal, way too extensive to cover on a mountain bike.  A downhill mountain bike is also gonna cost about the same as a decent trail bike.  Riding the track is fuckin fun too but I've lost interest in it as I've gotten older, is really expensive, can't afford to get hurt anymore, etc.
Dirtbiking is funner to do solo than mountain biking imo, but if you have friends that mountain bike that's sick.  Used to mountain bike a lot and loved it, lots of concussions though.
They're pretty different sports to try and decide between, I'd say a good compromise is to buy a trail bike like a KLX400 or something instead of going full motorcross