Not knowing what you mean by the 'entire panel', did you replace the resistor pack above the passenger-side kick panel that controls all of the lower speeds? Only the High fan speed doesn't run off the resistor pack.
Looks like this...
It is located up near the fan motor. If you see photos of a different type of resistor pack with a steel shield around it with coils inside, that's the older style which your '04 doesn't use.
A forum community dedicated to Jeep Wrangler owners and enthusiasts. Come join the discussion about reviews, performance, trail riding, gear, suspension, tires, classifieds, troubleshooting, maintenance, for all JL, JT, JK, TJ, YJ, and CJ models!