Sounds like you just need a recreational Kayak. Expect to pay between $500-800 depending on brand, quality, and features you want on it. Paddle and Life jacket are necessary.
I'd recommend buying used from someone. Im sure craigslist has something cheap.
But If your thinking about getting into trips or more technical Kayaking, look for a kayak with a Keel. This the type that makes you be able to rolls, tricks, surf, all the other super cool Kayak stuff. If your going this route I recommend taking some Kayaking classes. Go through ACA if your in North America.
If your looking for more information on the technical side of kayaking, PM and I'll send you some more info. I've done a decent amount of sea kayaking and have loads of information.