I have three toddlers and they love riding in our 4 door wrangler. It is indeed a capable "family" vehicle.
In terms of offroad mods, I don't believe there's a simplier or cheaper new vehicle for mods on the market today. You have endless choices for roof racks, bumpers, fuel cells, CB radios, blah blah.
The only problem for you is the tow ratings. I"m not sure why, but Wranglers have fairly low ratings given their power. Perhaps it's the high center of gravity for all models as well as the short wheel base for the 2 door Wranglers. Make sure you figure out how much weight you'll be towing and then look up the applicable numbers for the Wrangler you're considering.