Not sure if you are just looking for the differences between the 04 and 06 Rubicon or if you also want to know what the difference between a Rubi and a X or Sport is.
I've got an 06 Rubi and a couple of my buddies have different years . . . I'm pretty sure one is a 03 and the other is a 04 or 05. We all have manuals and the biggest difference is my two buddies with older Rubis have a 5 speed and I've got the 6. They say that the 5 speed is a bit more durable offroad, but the 6 speed is much better onroad. There are a couple other minor details like I've got plastic covers on the tailgate hinges vs their painted ones and they're entire roll cages are painted where as Jeep got stingy with the paint in the later years and only shot the areas that can be seen . . . fu@king bastards! Other than that, nothing comes to mind.