The tracking adjuster will tilt one wheel in so's you can start putting the belt on. After it gets about halfway, you will need to adjust that the other way and just bump the switch a bit a few times as you adjust and it'll work its way back on.
Take the collet fully apart and clean out all sawdust and other debris. Does this happen on all i/4 shank bits? use only quality bits preferably American bits. I use Whiteside and Amana, Cmt is a good bit. If it does not hold any 1/4 bit replace collet. To my knowledge Sears has never made a router, they just sell them. It's possible it was a Skill, a Black and Decker, or some other common brand with a few specs that Sears had them made with (good luck with that). Check router forum.com
other people have looked for collets, there is many suggestion or answers. t site has them, but it does not specify 1/4 or 1/2
OEM Craftsman Router Collet Asy 2823121000 Ships Today Fix com
Is it possible to grip each part with a pliers and twist and pull at the same time to release the bit? Let some oil run into the collet before you do this.