We'd be far worse off.. Religion offers a hope and a optimism that is absolutely essential for mankind to thrive. It gives our energies focus and helps us make sense out of life. It'd be tragic to lose that very important aspect of the human condition.
The problem isn't religion, its those who practice it (it's very much a 'guns don't kill people, people kill people' type of thing). For a great many 'religious' people, religion isn't about god. It's about a great many worldly things... Islamic extremists share a common trait with extremists of every ilk (be it Christian, Jewish, Buddhist, etc..). They are about control and power. They use God as a convienent stepping stone to what they really want....a world in which everyone acts and thinks exactly as they do.
That's not the fault of religion..it's the fault of broken people.