First a few notes: skeletons do burst out of the background at you during the first Mode7 stage (the one that I've thought perhaps the creators intended to be a fallen, rolling tower interior), and when you are younger or careless, those medusa heads in the second Mode7 area can easily knock you to your doom in the spike pits below. Mode7 still tickles me today just because how visually wild a concept it is and how rare it is to see in 2D games. (Bloodlines uses some variation of the tech interestingly, as well).
Now, one thing neat about the care of IV is the "layering" of the levels' art (sometimes you even go into the background). But basically there are all sorts of things happening in the foreground and background, whether its simply pillars leading up to the castle entrance or ghosts in the treasury. The levels feel alive with lots of movement, like the rising fence/gates in the first level. I don't think SotN did a lot of that, as far as foregrounds and backgrounds interacting with the main playing field.
And as for atmosphere, I point to the build-up to the finale. The fight with Slogra, Gaibon, and then visual presentation of Death and his room, with the alteration of the music, and culminating in the stair climb is just chilling. (Slogra and Gaibon were then turned into basically jokes by SotN and further entries). I think this game's art direction (IV) stands the test of time, personally.
Granted, the 8-direction whip did make the game easier, but I think IV had areas that really highlighted the usefulness of such an ability and made it play into the game design. (Perhaps you'll be pleased to know that Bloodlines, Rondo, and Dracula X decided to limit the whip again to one degree or another after IV).
The old games are the opposite of the newer games who tend rely on excess. I would say it better suits the mood of a lone man against an entire countryside of monsters. To brave them all, a warrior has to be precise.
Exactly. As was noted, the newer games are getting more and more "floaty," which isn't any more unrealistic that the cement shoes--in fact, it's more unrealistic and feels goofy (and less strategically precise). It mirrors the game design, which is also more meandering and floaty than "exciting." All the hit points stuff makes it feel less like I'm feeling the hits on enemies. Also, Alucard's walk is fine, but it always seems like a forward Moon Walk to me, and in the intro, his run is just the same as his walk, aided by the speeding background. (I don't care; just saying).
I agree about the 2D vs. 3D and aging...and yet, I can often look past that for the sake of the depth, art design, or ambition of a project. Heck, all that graphical fog in CV64 HELPS the mood!

It's the same with the historical limitations of the early games. I know what they were trying to execute, and for the time of their execution, they more than passed the test. As for tile sets, Dracula X and Bloodlines largely got rid of that while Rondo generally kept it.
********
EDIT: I wanted to add that, maybe I'm crazy, but the size and bold look of older sprites like IV's Simon and Richter look a lot better to me than anything from the GBA/DS era, which always frustrates me. Rebirth's Chris is in the ballpark, but the rest seem too understated and small to me.
And don't get me wrong, SotN is no slouch for a game, but I think it was best as a one-off, and certainly should have not continued beyond Aria of Sorrow, despite Dawn's improved graphics and Portrait's somehow amusing adventure. (I don't know what to say about OoE, as I have not played it yet). I think the largely unexplored Simon's Quest style has more room to grow than the Metroidvania style.
Anyway, I think Dave Cox said something important when he said gamers get to know too much about games before hand, but also it should be added that new games don't often provide enough opportunities to bring one's imagination to the game. That's more important than straight realism in my mind.