As far as safety, that seems to be subjective... so let me ask it this way: If removing them makes the vehicle flex more, does leaving them make it flex less, i.e. provide more stability at high speeds?
Yes, there is a safety concern operating on hard surfaces at 55mph plus making sudden movements, like deer dodging. That said, the increased length and width of the YJ (vs CJ) made it much more stable anyhow. Because of this, mixed with the fact that the axle doesn't really need lateral support, most folk think running without em is no different than with, except it flexs more.
There is nothing that is all good at no cost. Nothing.