It could be a stuck transfer case switch. When that happens, vacuum is always applied to the 4x4 light switch, so it stays lit even when you go into 2WD.
It's easy to check if you have a vacuum gauge. Look on the passenger side of the front axle and you should see a housing with 2 vacuum lines running to it. One one these will show a vacuum, one won't. As you shift between 2WD and 4WD which line has the vacuum changes. Disconnect one of them and hook it up to a vacuum gauge, and start the Jeep. If you see a constant reading as you go between 2WD and 4WD, then the transfer case switch is bad.
My 1994 did this, cost like $20 for the part at autozone. After I pulled my old one out I got the switch unstuck by hitting it with a hammer, but every few pushes it would get stuck again.