You can try to ween yourself off the caffeine but the best bet is just cold turkey. It hurts like hell for a few days but eventually it goes away.

Like Lil said though if you're going to take something for the headaches pay attention to the ingredients of the medicine. A lot of your OTC pain relievers have caffeine in them to help speed the action of the pain reliever. So no Exederine for sure and I think some Tylenol products have caffeine in them.

As far as the headaches being debilitating I know what you mean. I feels like some has wrapped a leather strap and around your head and is tightening it like a tourniquet.