Bad memory can cause all kinds of strange issues, from periodically locking up to the stuff you're describing. There's a free utility you can try here:
http://hcidesign.com/memtest/
Let it run for a while. 20 minutes minimum, overnight is best. If it keeps running your RAM is fine.
It's also a good check of your cooling system. It doesn't turn the heat up as much as other programs could, but if you're having problems like that I don't think you'd want to push it as hard as you can. Be sure to clean out any dust before you run it so you're checking the right things.