Well, the early segments had some difficulty, the clock tower certainly hurt me a lot (I'm thinking in terms of a first playthrough, not one after you know what you're doing and know the exploits), the approach to and fight with Scylla had some rough spots, the succubus can be pretty nasty if you don't know the trick, the chapel can whittle your health down (though the boss is easy), the arena has some rough rooms (although again, the bosses aren't too bad), Olrox isn't an easy fight, and Legion/Granfalloon, if you don't cheese it to death, becomes quite vicious.
That said, the toughest part for me in Inverted Castle (not counting Galamoth) was the alchemy lab. The particular types of enemies are a little harder to dodge in the vertical segments, and Beelzebub is a pretty rough boss.