Jerry Bransford· Knows a couple things...
2004 Wrangler TJ, 42RLE auto trans. F/R lockers, 5.38 gears, Warn Winch, Currie 4" Suspension
As Tangofox said, go for the brake booster hose. Having either end of that hose disconnected will result in your exact symptom. This is the large diameter rubber hose that connects the power brake booster (on the driver's side firewall) to the intake manifold where it gets its vacuum which powers the booster.