The sound quality gets real bad after playing for a while. If I take the cartridge out and put it back on it sounds fine again for a while. Should I just replace the cartridge or could it be in the tone arm.

Assuming it is not just accumulated fluff on the stylus or a worn stylus I would suggest you clean the contacts on the back of the cartridge - try a little squirt of switch cleaner (available from electronics stores or online from places like Amazon) If that doesn't cure the problem replace the entire cartridge just to be sure. All styli should be replaced after no more than 400 hours use but I normally just replace the whole cartridge anyway.

Are you sure it's not the amp it's connected to that's not the problem? Magnetic cartridges have to be amplified by a pre-amp inside the main amp. It's sole purposes is to do just that, so the rest of the amp can work just fine. Most of these pre-amps are just a single IC and a few other parts. From what you describe the IC is perhaps getting hot, due to some internal fault, when a signal is applied to it. Break the signal and it cools down making it sound normal. There could be one in the turntable itself, but I doubt it.
If you are good with a soldering iron you will find the IC by tracing the wires back from the turntable imput socket on the amp. Conecting the turntable to another amp will confirm it either way.