I suggest you chase Testosterone round the moons of Nibia and round the Antares Maelstrom and round Perditions flame before you give up.
I've wondered the same thing Khan, hit 40, picked up the spare tire, tired a lot, sex drive is "meh" etc. But I HAVE noticed that when I work out regularly, after a week or two, suddenly all I want to do is pounce on my wife regularly. So it's probably not a good sign that your work outs aren't making any difference that way.
That was my very helpful .02 worth.
"LOW TTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTT TTTTTTTTTTT!!!!!!!!!"