HOW TO HOOK UP SDAT 8

"Is there away to check the subwoofer output signal , to confirm if the signal is correct." Yes.
(1)Input 1KHz sine wave downloaded from internet into subwoofer.
(2)open the subwoofer
(3)connect an oscilloscope to the speaker output.
(4)Probe to +ve terminal. -ve terminal to earth clip.
(5)Observe any discrepancy.

Speakers don't connect to inputs. Have you looked at the manual?

Rear panel...
http://www.yamaha.com/yamahavgn/CDA/ContentDetail/EnlargedImage.html?CNTID=202147

http://www.retrevo.com/support/Yamaha-HTR-5740-Receivers-manual/id/1075bh240/t/2/

Consider the speakers as major contributors to the 'sound'.

The Yamaha can only drive powered subwoofers. Make sure that the RTD250 has it's own amplifier built in, you won't be able to easily run it off the yamaha if it doesn't.( the powered subwoofer will have a mains cord to power if it's correct) If it doesn't have a amp built in, easiest weay is to buy a cheap powered sub.
Is the RTD 250 the sub fromthe RCA home theatre pack?

It sounds like you are describing an earth loop problem. I won't go into detail in this post, but try an isolation test first. disconnect everything from the reciever except the sub and try connecting say a BATTERY powered mp3 player or cd player to minimise connections to the electrical mains earth. you could also isolate things by taking your sub to a friend's house and trying it on their system.
If it is an earth loop use a process of elimination to find out which two pieces of equipment causes the earth loop. then apply filtering.
you can buy earth loop isolation transformers for about AU$26 and I've used one on my sub myself. Theyhave RCA (phono)plugs on each end and they connect inline with the signal cable to the sub amp.

as for the thumping sound it might be the speaker protection circuitry cutting off the amp to protect the subwoofer speaker from damaged. This might be triggered if the earth loop sound is causing the sub amp to overload and clip.

Failing this there might be a dried out overheated electro capacitor in the amp circuit causing the hum. Hav a chat to your local electronics person, who should be able to spot it!
Hope this puts you on the right track
Happy Hunting!

my phone say that initial programming needed can you sold this for me

You can use Optical SPDIF cable. Both yamaha and xbox supports it. You can find cheap optical cables on ebay ($1-5 max.)

You can use a hi-level to low-level adapter. Just remember, the Yamaha does not send power (wattage) to the sub, just signal. You need to use a subwoofer that has an amplifier either externally or built in.

as long as there is a speaker out from the receiver you should be ok just make sure polarity for spkr is right --remember you workng with AC current bad wire will cause fire==

IF ITS A POWERED SUB,,JUST USE RCA JACKS TO THE SUB TO THE RECIEVER VIA THE PLUG THAT SAYS SUB

sometimes these subwoofers are for builtin amplified subwoofers..so if this is the case it will not work