treat your jeep the way you want and just try to ignore those comments. i personally don't baby my jeep, don't wash it as often and don't keep the inside as clean as possible. however i don't do any serious wheeling either as i'm afraid to break something i can't afford to and this is my daily driver.
on the other hand i have a friend who inspects his mazda 3 everytime he walks up to it and rubs off any smudges of dirt that he finds and it drives me crazy how much he babies it but hey that is his car.
to each his own. whoever paid for the jeep gets to make the rules.