Took my scout troop here in the spring of 2016, after many visits on my own and with my family. We did the hike from Old Man's Cave to Ash Cave, and one of the boys said "It looks like Middle Earth!" While the easy-access areas are cool, you owe it to yourself to get off the beaten path a bit. The crowds are much smaller and the scenery is even more wonderful. I also strongly recommend the Conkle's Hollow rim trail, especially in the Fall. Be careful with little ones, though, because there are a lot of drop offs, and it's a long way down.
Also worth mentioning are North and South Manitou Islands. You can do day trips to South, or camp in one of four campgrounds on the island, but you have to carry all your gear. North Manitou is much wilder and is more for backpacking, but there is also a campground near the dock. And both islands have awesome beaches all around.
Be sure you wait until the water is calm before you head out to it. It's covered with barnacles and if you get washed up against it, you could get torn up pretty bad. Here's a closeup I took:
http://twosheds1.deviantart.com/art/Pocahontas-Shipwreck-407297276