That elbow is commonly referred to as the 'trumpet'. Its sole purpose in life is to speed up the air flow so the air intake is quieter. That's the only reason it is louder when you remove the trumpet, not that the engine is getting any more air since the trumpet easily flows all the air the engine is capable of consuming.
Even last month's Jp magazine confirmed via a dynometer test that removing the trumpet does nothing good for the engine's performance. What removing it will do is give water easier access into the engine since that is the trumpet's second purpose, to help keep water out of the engine.
I'd put it back in. Jeep's engineers designed the STOCK air intake system to easily flow more air than the engine is capable of consuming. No kidding, no rumor. This is right from the horse's mouth, so to speak... Senior Jeep engineer Jim Repp (known as the Rubicon's creator/"father") has confirmed that several times over the years, including in a personal conversation I had with him 4-5 years ago when we were talking on the phone about something else... I asked him about the air intake.
SOME engines benefit from freer breathing air intakes, like the Mustang 5.0L and Camaro Z-28, but not the 4.0L since its air intake is definitely not restrictive in the least.