Mine was doing that, and it was a vacuum leak. There is a vacuum hose on the passenger side connected to the firewall, close to the battery, find it and follow it till you find the leak. Took me five min to find and fix it once I knew what I was looking for.
As others have said, its probably a vacuum leak. My 98 Sahara had the same issue when I bought it. A mechanic spent all kinds of money on new switch boards and other stuff. None of it worked. Than he popped the hood and there was the vacuum line just hanging. It appeared it had contacted hot metal and burned completely through. He replaced that cheap tube and all worked fine.
Had the same issue with my 2005. I nearly tore my whole dash apart looking for what was causing it before I did some easy research on the net. It's the vacuum hose. It's come loose. On my 2005 the hose was attached to the drivers side engine. More towards the window than the front of the engine. It was about 8-10 inches from the back of the engine (window) and about 4 inches down from the top of the engine. Small hose just hanging next to where it was suppossed to be attached. I used a zip cord, reattached the hose and zip tied that sucker down hard. Hasn't come off yet. Ias mentioned the hose is attached to the passenger side fire wall, goes down under the battery and winds up on the engine. Once reattached all vents working. Took more time following the hose than reattaching and cost nothing to do. Good luck.