If you can afford it, buy the Rubicon. If for nothing else, resale value. Nicolas is right though, a basic X with the right tires will get you by, but think of the Rubicon as a total complete no fuss package that you would have to do very little to, if anything, in order to wheel where you want to wheel.*
*Disclaimer - No Jeep is completely no fuss. There will always be something that you will want to add to it.