The castle is not an earthly castle taken over. Multiple games have proven otherwise. Harmony of Dissonance - "A castle not on any map" had appeared out of the fog. SotN also states similarly. The castle appears and disappears. Even to the point of two versions of itself, ala SotN. I believe even PoR states that the castle simply appeared.
That said; Some titles suggest that Castlevania is rebuilt and destroyed as different castles. As we see in Castlevania II, and Curse of Darkness, the ruins remain. I believe Bloodlines also has this concept in place for the first stage.
So it would stand to reason, that either Dracula creates a new castle entity every time, or the entity that shapes the castle, is directly fueled by Dracula's magic to construct the physical castle, but once Dracula is out of the picture it cannot keep the physical castle together and waits for another suitable magic source. It forges a new one every time Dracula links up his powers again, and cannot sustain it without his presence, unconscious or otherwise. Either way the castle has been proven to never stand without Dracula's immense power being present behind the scenes. Even Maxim was taken over by Dracula's remains which fueled the castle, and Brauner had to seal Dracula but draw his power to sustain the castle's form.
As for LoI's castle? Was that even Castlevania? I don't believe there was any indication it is the same. I believe it was just a castle Walter either had built, or took over long ago. Did it even crumble at the end? I can't recall. This raises the question of, did the castle exist before Dracula, or did Dracula create the entity that is what forms the castle.
The castle itself being a sort of living entity is really cool in my opinion. It both serves a game mechanic of never the same map, while also being a unique and interesting explanation in itself. In the way that the castle itself is alive with traps, and is your enemy, it makes sense.