It should be disconnected for off-pavement use, if only for comfort (less of you getting shaken side-to-side over bumps). If you opt to do disconnects, the JKS Quicker Disconnects are a real pleasure to work with.
Discussions about removing the front swaybar entirely usually open up a can of worms, and many will advise against it. Ultimately this is a personal choice - either you like the way it behaves without a swaybar, or you don't, or you do but you're weirded out by the idea of not having a "safety device" in working order. As a personal choice, and I won't fault someone for making a choice that works for them, whatever that choice is. The rear swaybar should remain intact, regardless.
For what it's worth, I left my front bar disconnected for 2 years before I finally removed it entirely the last time I had my winch plate off. I routinely put on 600-1000 months per month for work, plus personal miles, and I like the way my TJ handles now more than I did before, especially over the crumbling roads here in Michigan.