Don't go down whatever you do. When down is compacted it doesnt insulate properly which will leave cold spots.  Synthetic puffy, merino wool, 100% polyester tech hoodie, or fleece will all work pretty well.  The merino wool will probably be the best though