I had the same issue, and it drove me crazy for YEARS! I even disconnected the panel vent actuator so it could stay open all the time. Having the AC blow up on defrost, and not right at you is horrible in the Florida summer!
Anyway, it is absolutely a vacuum leak. I tried replacing the check valve and the vac line that connects to the manifold, thinking that was the culprit (letting vac escape back to the engine under load). Then I took out the battery expecting to find a problem with the vac reservoir, since that FSM says that it the part that allows the vac system to work when under load, and found no problem. After some testing with a vacuum pump, and gauge, I found a really fast leak in the "half open" side of the defrost actuator. The one with the yellow line connected to it. I pulled the line off the actuator, capped it, and now everything is perfect except the defrost is always half open, or full open when actually on defrost. I will grab a new actuator next week, and be good as new.
Do yourself a favor, get a vacuum pump and just test each section of the system (I borrowed one from the local Auto Zone). It took an hour to find the problem once i had the pump. Without the pump, I would have tried replacing the reservoir, more lines, and probably the wrong actuator, since it was the PANEL that had a hard time staying open.