Who is claiming special interests have the welfare of the American people? They don't. They care about their special interest. That's why they're special interest groups. Every group does this. Labor doesn't care if their industry is obsolete or if the country would be better if their factory closed. Pharm companies want to make money and charge a lot. Abortion activitists want their moral views forced on everyone else. Their whole existence is to champion small causes - regardless of the ethics.
I would argue you need special interest groups or you have majority tyranny.