Well, with the actuator ruled out, and the fact that you're getting vacuum to at least one of the lines, that leaves two possibilities. One is a broken/leaking vacuum line from the vacuum switch on the transfer case. And the second is that the vacuum switch is bad.
You should have vacuum to the pass. side of the actuator in 2wd and to the drivers side in 4wd.
You're going to have to check the lines that go from the actuator to the switch for breaks and leaks.
If everything checks out with the lines, you'll need a new vacuum switch.