Another vote for steering not being centered. If it is, maybe mud on the inside of your wheels, or a clockspring problem?
Clockspring issues crop up here from time to time. Usually causes ESC or ABS lights to activate.
Even if your caster is into the dreaded negative numbers range, that shouldn't by itself cause ESP probs.