I have that problem, too, but on a much smaller scale. I usually wear my glasses in the morning before showering. Then I put in contacts after. I'll wear my glasses if my eyes are bothering me, allergy overload, or if I'll be reading a long manuscript for many hours.

I can't swap from glasses to contacts without giving my eyes a bit of a rest without anything. The shower is a good rest period, but even a few moments with my eyes closed makes a difference. Otherwise I get the dizzy, unable to focus thing, too.