Because they soak up water, wick it up along your back, and freeze. When cotton is wet, it has absolutely no insulating qualities, and it gets wet all the time when you're skiing.
Tall tees look decent on people who can kill it in the park, but tbh they look absolutely ridiculous anywhere that isn't a man-made jump or a rail. Look at Bdog and Ahmet's parts in Level1 movies- they look good on rails and stuff, but they look downright retarded skiing pow and hitting cliffs in a tall tee.