Personally, of those years I would go with an '03 or '04.
-'03 up had better interiors
-'05 and '06 had "laughing monkey issues" (see more info in link)
2005 and 2006 TJ/LJ's... OPDA (barrel of laughing monkeys) and reimbursement write up
Also, if getting a manual:
-'03 & '04 had the 5 speed, vs the 6 speed in '05 & '06. The six speed isn't a bad trans, but it's harder and costlier to replace. While the 5 speed has better aftermarket support and is much easier to find.
Lastly, look for a Sport Model with the 4.0L and a Dana 44 rear. FYI, all "X" models only came with a Dana 35 rear.