This problem is far more common in TJs than should be, that is for sure. Based on your symptom, it is easily fixed by replacing your fuel pressure regulator which sits outside of the gas tank on top of the fuel pump.
When the FPR leaks, it allows the fuel that is supposed to stay in the top-end of the engine (fuel rail that the fuel injectors plug into) for fast starts to instead drain back down into the gas tank. The extended crank time until the engine starts is how long it takes the fuel pump to get fuel back up to the engine. Turning your ignition switch on-off-on a few times to cycle the fuel pump before trying to start the engine will help shorten the start times until you can replace the FPR.
Once the fuel tank is dropped, it takes less than 5 minutes to swap the FPR which simply snaps in & out of the fuel pump. I'm holding a new FPR in the below pic, you can see the old FPR below on top of the gas tank.
RockAuto Auto Parts
is a great source for them. Rock Auto's pricing and delivery times are hard to beat.