Ok, let me try to explain what that meter is doing under particular tests, it may help.
Basics of electricity: voltage is pressure of electricity, resistance is restrictive force (think corrosion), and amps are actual power flow. In a water hose, amps would be how much water is coming out. Volts would be how much pressure is pushing the flow, and resistance would be how much mud & junk is on the inside of the hose. So if you turned the hose on, and very little water came out, you would have low amps caused by high ohms OR low pressure - time to test for voltage drop to see which. If voltage is normal, test the circuit for resistance, because you have good pressure but not flow - it must be restricted somewhere. If it was flowing normally, and you put your thumb over the end, you would be increasing pressure (voltage), but decreasing total flow (amps).
An open circuit is a "broken" wire, a closed circuit is a solid wire. One has power flow, the other does not.
Watts are a way to look at total power consumption without reference to specific amp/voltage. A circuit of 12V, 15A, is equal to a circuit of 120V, 1.5A. They would both use equal watts, 180. (W=A x V). Watt is just a measure of power used, like a Joule(?) in physics.
When the setting is on resistance, or Ohms, the meter tests the conductivity of a circuit or wire. The way that it does this is by sending a tiny pulse of electricity out the pos lead, which in this case you had attached say to the coax or mount. Then, when you place the neg lead on the frame, say for example, the meter takes a reading of the remaining pulse that it sent (having travelled from the mount through the tailgate and hinge, down to the frame and into the neg lead), and determines from how much energy made it back on that neg lead how much resistance is in that circuit. If you did this to a live circuit with a load, like a light bulb, and tested both sides of the light, the power for the light would have an easier path going through the meter than that light, so your meter becomes the light instead. If the light uses enough power, you will smoke your meter doing this. It is the exact same as taking a bare piece of wire and connecting it on both sides. This is why the circuit must be off. MUST BE. With house voltage, getting this wrong can kill you, and at all voltages will damage meters, componants, and circuitry.
Relavent to you - With your meter hooked up and the radio on, instead of testing the conductivity of the mount to frame ground path, you are providing an alternative path (a ground strap) through your meter. The reading is how much energy is travelling through those leads to find ground. Because it is a ground circuit, you have'nt bypassed any loads, so the meter displays instead of frying. BUT there is other energy in the meter aside from the pulse it sent out, so it just gets confused on whats happening.
So how to test a live circuit for conductivity?
A voltage drop test. While set on voltage (must be ac for ac, dc for dc, but they work the same), the meter does something totally different. Instead of being a closed circuit, it is open, not allowing power to flow through the meter as it does in a resistance test. The meter looks at each lead independently, and displays the difference in voltage between each two. So for your light bulb test, with the light on, you would connect one lead in front of the light (pos) and the other behind it (neg). It would take a voltage reading before the light, then the light would use whatever power it wanted, then the other lead would show the meter how much voltage is present after the light, and it would display the difference of those two for you to see the consumption of that load (light). Wih this test you can actually stick one of each lead into an electrical socket (plug), just like mama told ya not to, and be fine. Try it with an ohm test and you'll loose your eyebrows when you start to glow, and will not be fine.
Relavent to you - When you put a lead on the coax collar (sheilding) with the meter on voltage, the pos lead will look at how much pressure is there. Then with the neg lead on frame or batt ground, it will look at the pressure there. If your path to ground is good, the pressure will equalize, like a hose with water layed out straight. If your path to ground is not good, you just folded that hose in half. Now pressure on one side is high, while it is low on the other. The result? No water, or very little, coming out the hose, because of a choke point. Then we can start ohm testing small areas to localize the "hose fold" or choke point.
Make any more sense? Not sure how good of an explination that is, it took me more than a conversation to get it. I still know auto techs that are lost with a multimeter, they are not easy to understand. Nor is electricity.