the foam itself does nothing to increase safety in a crash, it's purely comfort. I've pulled mine out because I always wear a beanie under mine so I get that comfort back, but you'd be pretty dumb to pull it out and have the foam layer directly on your head.
that being said though, a firm fit is still important, ESPECIALLY if you're someone with one of the MIPS helmets, where you want the inner layer firm on your head, but so the shell can move around like designed for shearing impacts, but either way you don't want a helmet that has excessive movement. honestly just common sense really