Some thrift stores will carry vinyl, like certain good will stores and shit, and most Oxfam stores. Other than that theres the internet, or a discount rack at your local record store.
Most discount records are going to be shitty old David Bowie albums and shit though, you won't find anything cool.
I actually think Hot Topic has pretty decent prices on Vinyl, but they will have like a Bob Marley album, a Sublime album, a couple decent punk/metal albums, and then pure shit. A bunch of them are picture vinyl too, I don't know how you feel about that.
I'd reccommend some mail orders bu they're all genre specific and I don't know what your looking for. If your looking for mostly punk/hardcore I can hook you up with some cheap mail order places probably, like Profane Existence for example.