I actually thought Dawn's castle was bigger than SoTN, to its detriment. PoR probably has more space with all of its areas combined, OoE too.
I mean, SoTN is big if you take in account the Inverted Castle, but on its own, it's at an optimal size I think. The Marble Gallery and Outer Walls aren't that big and yet act as sort of main hubs for the rest of the castle's connections and flows greatly because of how it's not extremely huge. Any of the excess isn't required to run through most of the time. For example, you don't even have to explore beyond the second floor of the coliseum to get the needed relic.
SoTN also had the least copying and pasting so that helps too.