Sounds like you'd have a great time doing something like the Mojave Road. It is a roughly 130 mile road that crosses the Mojave desert and typically gets traveled over the course of 3 days. It can be done in 2, even 1, but there are a lot of stops you make to see interesting sights throughout the entire trail.
You might see a few other travelers there, but for the most part, you're away from it all deep in the heart of the desert. The trail varies from light sandy area, to hard packed dirt, from low elevation to high elevation at around 5000 feet (have woken up to snow on the trail before, like lots of it). There are tons of old abandoned cattle ranches, and even a lava rock cave area that is interesting during the day because of the way light enters the cave.
Joshua Tree National Park - Go to Cottonwood Entrance - continue 6 miles up past the Rangers Station to Old Dale Road - continue North just past Mission Well to Brooklyn Mine Jeep Trail (Signed entrance) - Take your Jeep all the way up to the mines and set up camp next to the old shack at the end of the trail.
Very quiet, and difficult to reach canyon with great 4WD Jeeping as well!
Gaggs Camp. Not kidding on the name. Way out of the way. Gorgeous- river runs through it. Ok- so a full creek but little trout swimming in it. Not many people and no amenities aside from vault toilets. Pristine. Sierra National Forest - Gaggs Camp Campground
Didn't stay- drove through it to check it out but weren't "pack in pack out" prepared. 40 minutes to anything.
A little known area is Bass Lake- just south of the southern entrance to Yosemite. You won't be alone here- but book 2 sites cheap and have some space. Plenty of places to explore plus you can rent a little patio boat for $150 a day- complete with wienie grill. The lake is pristine, clean and Angel Falls fed.Perennial favorite- just not on a holiday weekend.
Highly recommend Foghorn Outdoors California Camping book. Has "5%" classfication for hard to access/remote/4WD sites. Great resource. Thousands of campsites with reviews and ratings. Foghorn Outdoors :: California Camping
Bass Lake. Almost hate to tell people about it so it doesn't get well known. I am in Corona too. About a 5 hour drive. Wishon Point and The Forks are both well maintained and convenient. I also like "remote" but have kind of given up on it in Southern California.