I just passed a guy on the highway having the same problem.
It's very common, and yep--it means you're doing it wrong.
Here's my standard answer. Post back with any questions.
Originally Posted by MTH
1) use Velcro to loosely hold window in place.
2) zip only about 4 inches of zipper.
3) fold the solid strip of the door-side of the window 180 degrees into the channel in the black plastic "door surround" that abuts the door when the window is out. The strip will literally have been folded over such that it wraps around the door surround.
4) zip the rest of the window.
5) pop in the lower edging of the window into the channel along the upper edge of the jeep. This is very much like what was done in step 3.
6) fix the Velcro.
7) admire own handiwork.
Window will be very tight. Nothing flaps in the breeze.