Next time try turning the key to ON for a few seconds then turn it off, then turn it on again and try to start. If it fires right up then it's some sort of fuel priming issue. When you turn the key ON, the pump will prime the lines.
Could be fuel filter (cheap and easy). Could be the pump though generally speaking, when a pump goes, it just goes.. And if it were pump I'd think you'd notice symptoms other times than starting..