The people who like the N64 games are a VERY vocal minority. Members of this camp will rant and pout enough to make you believe the games were actually liked, if not loved.
You are entitled to your own assessment of the games but I must question this point. I have not seen someone make such an argument, and I think for the most part we are merely individuals who like to form our own opinions, rather than go into something with a negative mindset because we're told to dislike it.
I know my contention has always been that the N64 games get a bad wrap because they failed to live up to expectations and standards to which they never aspired. I've not claimed the games were successful or generally well liked. Critics and players who only knew the series from SOTN (i.e. "nearly every gamer") were disappointed that the 64 games were not MetroidVania in 3D. Even though I'm pretty sure that would have been impossible on the N64, and IGA's own 3D entries have shown us it might not be the best idea anyway.
The 64vanias did not reinvent the wheel as Symphony of the Night had done. In terms of gameplay they were translations of the ClassicVania formula into 3D, nothing more and nothing less aside from some character and plot fleshing missions and the inventory/shop system. Artistically and in terms of storytelling they broke (or at least stretched) the mold in ways I found refreshing and personally I think the artstyle is beautiful. The camera in LoD is also a matter of taste, I was bothered by it a grand total of maybe 3 times in the whole game, being totally honest. Which is far less than in a lot of games I've played, including Lament of Innocence.
All that being said yes, there are and were a lot of people who like the N64 titles, just because IGA or anyone else doesn't like them does not make our opinions automatically invalid. And it's a dangerously easy step in implication from "vocal minority" to "minority so it doesn't matter", especially when we have no way of knowing exactly how big this minority is.