It will technically work, but it'll be baggy. Especially since you have an 03-06 with the larger padding. The top will have taken a set to your Jeep with the thick padding and will be sloppy without it. If you try it, just make sure to save the padding and take it off nicely so you can put it back on if need be.