1996 RAV4 pops out of 5th gear but if I hold it in it fine. All other gears are fine as well. No brass or metal on gear oil

Most likely it's missing transmission oil. Instead of holding it in, he should have checked the level of oil and added some if needed. Now the transmission is probably damaged and the repair will be much more than just adding oil. Sorry.

Unfortunately it sounds like a transmission rebuild. Manual transmission use syncronizers to allow the gears to mesh correctly when sifting. When it "pops" out of gear, it may have been a bent shift fork or a chipped gear, but now it sounds like it's sheared. You can remove the drain plug and see if the oil is contaminated with metal shavings to confirm.

if it is grinding i would say it is the synchros

converter came loose

baulk ring in gearbox needs replacing.(otherwise you need to hold gearstick in 5th)

You probably need to rebuild the transmission, and look for gears that have been damaged, synchronizers that have failed, and worn parts in general.
The clutch has no bearing on whether the tranny slips out of gear.

trans needs overhaul. 5th gear/syncro is worn out.

This Sounds Like The Brass Clutch Ring For 5th Gear Has Broken Thus Without it It will Not Have Pull in that Gear This A Problem that Calls For Removal and Rebuilding of Transmission.

there are dent and brass balls that are in your transmission and are used to hold the shifter into each gear when it starts poping out of that means that those dents and brass balls are worn down and i would suggest getting the transmission rebuilt.

You need a new 5th gear- luckily this is easy to replace- I did it myself following online directions for 5th gear replacement. The 5th gear is not in the box with the other gears- its off to the side so you can access it very easily by removing the left front wheel. Just don't let the idler assembly come apart, lots of tiny parts and springs...
The 5th gear can ware out prematurely if you rest your hand on the shifter- do not rest your hand on the shifter in the future.