I just downsized from a '13 Ram Hemi Crew Cab to a '12 JKU Sport. It's my DD, and you'll be fine.
I have three boys, 14, 12, and 9, and while there is the occasional outburst (mainly from the 9 yr old as he has to sit in the middle and can't see out the sides), it's fine. And my 14 yr old is 5'10" already......I made him sit behind me (I'm 6'1") on the test drive to make absolutely sure there was enough room for everybody. It is a tad crowded, but the Jeep is about 2 feet skinnier than my Ram or your Burb.
We driven it with a fully loaded pop up tent trailer (loaded with gear, and kids, soft top, doors on) about 3 1/2 hours one way, and the kids (and us) survived.
As for the daily driving....I find any excuse I can to get out and drive it!
It sits nice and high, (no difference I've noticed from the Ram), and if you have a soft top, oh, yes, there will be road noise....
And as for the gas, you should save quite a bit over the Burb. I went from 20L per 100km in the Ram in town, to 14L per 100km in the Jeep. And I'm not nearly as gas conscious with the Jeep as I was the Ram. Up here in Canada, gas is about $1.10 Cdn for a litre, so for every 100km I drive, I save myself about $6 just by switching to the Jeep.
As well, my insurance dropped $40 a year by going to the Jeep from the Ram.
Just another reason for you to pull that trigger.
Totally unbiased here, but.... DO IT.