I used to have a 02' Subi WRX and now have an 01' Nissan Pathfinder. Both similar cars to the outback and xterra respectively. Unless you NEED the big back seat in the xterra, go for the outback. Better on gas, more reliable, arguably the same amount of space for stuff (with the exception of the back seat), and fantastic AWD. I love Subis, and will probably get another one in the future.
Why, do you ask, did I sell the WRX and go for the Pathfinder then? The WRX I had was lowered and had an STi suspension. Maybe I'm an old man, but it was getting to be too stiff for me. Also, very small backseat (way smaller than the outback) and I could take clients around in it. Plus I got a screaming deal on the Pathfinder and the change between the two cars put an extra $10k in my pocket.
A year or two from now I'll likely get an 06' or 07' legacy 2.5GT wagon or outback 2.5XT. Or possibly an 05' allroad if I feel like pissing money away on repairs. lol.