When you take them off, drive around, freeze your butt off, and come home and say "I'm never doing that again," it's too early. Everybody has a different tolerance/preference for temp and wind. Go find out where yours are. Worst case, you'll have a good Jeep story to tell next time someone asks you this question.
How soon is too soon to take the doors off?
