Couple of things, excessive negative caster will cause what is called an egg roll. Feels like the Jeep is trying to stay balanced on top of an egg. Negative caster to a point makes steering easier but if you go to far, you end up with a shopping cart. Where it want's to follow every dip/bump/roll in the road. Going slightly more positive on the caster will make it run straighter and not tend to follow the road as much.
Also did you check for play in the steering box? Actually too loose or too tight of steering box play will cause it to wander. Too loose, and the Jeep is given free rein to do what it pleases, and you are constantly playing catch up to it.
On the opposite side, too tight. We used to deal with this all the time when I worked for Lincoln/Mercury. Folks would come in complaining the steering had too much "play" and they were constantly having to correct it. In actuality the steering box play was too tight, so any minute movement of the steering wheel was met by an instant reaction from the vehicle. And again folks were constantly chasing the wheel as they drove down the road.
Simply loosening up the steering box a little and it took complete care of the problem.
97 Sahara, 74 Renegade, 72 CJ5, 55 Willy's (the Beast) Boss 302, 6:34 gearing, Bose 901 speakers.
God I miss that Jeep. Forgot the kinda Jeep, 80 SX4.