I often skip/put off the extra modes/hard modes too, but the ones in Circle of the Moon are worth doing, in my opinion. I've actually done all of those. It's less of a time sink than it seems. After the first time beating the game, I was blowing throw the extra modes in around 5 hours apiece. Each one changes the experience of playing the game, not just makes it harder, so it kept it interesting for me.