2004 Volvo S40 Battery Removal

Use a multimeter to check the current (voltage and amps) being generated by the alternator with the engine running (at different rpm's). Also, after fully charging the battery with an external charger, check the current and see if the car starts fine without issues.
This could be the alternator grounding and draining energy.
This could also be a EPM module if there is one present.
I've seen those sort of issues with a 2004 Ep3 civic (EPM faulty), a 2004 S40 volvo (alternator faulty) and a 2004 s2000 (battery faulty)..2004 seems to be the "built with electrical issues" year in my book.

this clip is for 2006 but it may help you out. How To Replace an Alternator on 2006 Volvo S40 T5 Part 1 Removal

Most have a recessed allen screw, or star-key nut located on the back side of the knob...some are hidden by a clip on cover...look for at the base of the knob design...

the second catch is a safety catch. reach in with your hand and move it to the side.

You will need to remove fuel pressure first,I will send you a link on how to remove fuel filter and to the left is a link to remove fuel pressure.

this is what holds the battery in place it usually twists one way or the other to un lock it and there should be a bolt perhaps at the base of the battery with a plastic type holder

remove gas cap to release pressure. Then buy the cheap plastic removal tool that slips on the line and inside the filter to remove it from the fuel lines.

There may just be one screw that holds, one end of the cover, then you can slide it off. But you will have to bend over backwards.
The fixings may be star bits, but other than that, just regular bolts and nuts.
Hope that helps.

did you remove the screws from the arm rest screws? from the top and from the side and they have plugs in them so that you don't see them