When Castlevania 64 came out, I remember playing it and thinking it was interesting and good, but I sort of took it for granted and didn't necessarily think it was special. In fact, I think Lament of Innocence had a stronger first impression on me in some ways. But I never thought of either N64 game as bad. And then, sometime after Curse of Darkness or the first Lords of Shadow, I really began to admire what Castlevania 64 (and Legacy of Darkness) had done. It wasn't simply nostalgia. They're flawed, but still managed to legitimately bring Castlevania into the third dimension in a big way.
I've found that over 90% of the camera problems can be fixed with the R Button, and I've never had much issue with the controls, especially after some practice (aside from a few ledge-grabbing sequences, I suppose). I posted a video earlier in this thread, and when you get in a rhythm with this game, it flows pretty darn well. The N64 game was the growing pains for Castlevania in 3D, but those growing pains were cut short in a "helicopter parent" sort of way with the attempts at taking shortcuts with LoI, CoD, and LoS. (i.e. Let's do Castlevania through the lens of Devil May Cry, God of War, etc.) LoD wasn't so much a full-fledged second attempt as it was a DLC patch that was likely trying to shore up development costs from the first game.
Everyone is entitled to their opinion, but like Abnormal Freak, I tend to give credit for the overall experience over the nuts and bolts. In today's overly perfectionistic world, we tend to judge video games by if they dotted all their I's and crossed all their T's, but if that were the case we could retroactively condemn a whole host of classic, influential, and fun games. And on that note, I'm trying to figure out what Castlevania 64/LoD is being compared to in that sense. The 3D biggies of its time were like Mario 64 and Tomb Raider and a few others. (You can throw in Zelda: Ocarina of Time and Metal Gear Solid...but those are not platformers by any means, and the former couldn't really be used as any sort of major reference given the overlap in development times.) Anyway, when you look at the general attempt at realistic proportions, physics, and environments in terms of translating real-time, input-driven 3D action-platforming with exploration elements and the quirks/trappings of the Castlevania franchise, it had a taller task than the highly praised Mario 64. If I compare it to contemporary games that I guess had similar-ish tasks...maybe Soul Reaver, Medievil, Nightmare Creatures, Tomb Raider...Castlevania 64 is generally less herky-jerky in controls/camera/visuals and in many ways just as or more ambitious while doing it. So, while the N64 era of Castlevania could be awkward or rough around the edges at times, for a first attempt, I think it stood up against the competition just fine and I don't completely understand where the judgment is coming from on that front.
I suppose there is the front that it wasn't SotN or SotN in 3D. But that's unfair. That's like saying Castlevania IV wasn't Dracula's Curse, or Rondo of Blood wasn't Castlevania IV, or Bloodlines wasn't Rondo of Blood, etc. Castlevania has always been a series that experimented and remixed itself, and the N64 games tried to bring a little bit of everything, including their own spin. (I guess I've heard the argument that the N64 games took too many cues from Tomb Raider...but that's not a fair argument, because they each brought a different gameplay mix to the table and one could also easily say that Mario 64 was too much like Tomb Raider compared to the franchise's earlier 2D entries.) There's a place for different interpretations within reason, where there is give-and-take from each entry, and that's what helped make this franchise great, IMO. If I'm disappointed about anything with the N64 games, it's that they weren't allowed to be completed as planned. The interviews and beta videos promised even more than what we got with the additional material of LoD. The hearts and minds of the people that worked on this were in the right place, I feel.
There's no denying that of all the 3D Castlevanias, the N64 ones still captured the spirit and gameplay of the originals the best. (Although the actual best 3D Castlevania is called Bloodborne.)
Not so fast...Bloodborne's got the visuals, atmosphere, interconnected world, and challenge...but it's lacking in the "platforming" and enemy placement departments, and has too much combat grinding, just like Dark Souls. I think we have to be careful that, should such a thing happen, the next 3D Castlevania doesn't end up trying to carbon-copy a successful 3D game that's missing fundamental ingredients of the franchise. It's probably much harder to get a potential new 3D Castlevania up and running with a paradigm of its own, but that's what the franchise needs to stand out and get the most mileage. I think Lords of Shadow proved that to me once and for all. But I doubt there's the guts to go for it at this point.