Low blood sugar or reaction to 'caffeine'?

Yours might be a caffeine allergy, but keep in mind, too, that if you're consuming a lot of sugar in the coffee, that could be causing your blood sugar to crash later.