This is a hard, but fun question...
Overall, it's a tight battle between Zelda II and Zelda III as far as I'm concerned...
Since Zelda II doesn't get a lot of love, and because it introduced a lot of things like the magic system and combat techniques, I'm tempted to give it my vote. Also, the amount of towns w/ two large islands gave it a sense of immersion that's hard to match. The soundtrack is highly under-rated, as well. Finally, I like how the game doesn't baby you, and you have to find a lot of stuff by just exploring or inquiring in villages. It really is an amazing Zelda game, and probably my personal favorite. Despite the graphical limitations of the time, it has a more mature atmosphere to it than any other Zelda, I think.
However...Zelda III was perhaps the most complete package, as it blended the best elements of 1 and 2 and put on even more layers. None of the games since have been so well designed for so long a quest (the map space was used very efficiently and creatively). As one prime example: VERY few Zelda games will you run across such useful, optional items as the Magic Cape, Magic Boomerang, and Staff of Byrna in such obscure locations/ways. The Ice Rod almost fits in that category, too, but you actually need it in the Dark World much later on. Anyway, Zelda III really rewards you for exploring, unlike later OOT and TP entries that have big field spaces, but nothing but rupees or minor sub-quests to achieve in them.
Link's Awakening is a strong third place winner for me. Fun, mysterious, and epic, it really made a cohesive island quest come to life. Who doesn't love the side quest where you lead the ghost around to bring it peace! Brilliant.
I'll also show some love for Majora's Mask, too. I enjoyed it when it came out, but it's only after playing the later main entries since that I've grown to realize how strong a game it was/is. The interaction with the town and how it changes is amazing. The darker, weirder feel was nice, too. By far the best soundtrack of the 3D Zeldas.
Wind Waker didn't do much for me. Too much sailing, too short a quest, not enough new and exciting things in that time to make it all hang together.
Now, I've got to just come out and say that Capcom's tenure with Zelda was essentially better than anything Nintendo's done with the series in the last 8 years. The Oracle games came out of nowhere and were able to more or less match the magic of Link's Awakening. Then, Minish Cap was so unique and immersive. It may have been on the short side, but it made EVERY MINUTE of it fun and engaging. You really got into the world as you explored it from two very different angles. The idea of it almost seemed beyond GBA power, but the game design and art direction made it work in a big way! Its soundtrack was incredible; so many cool new tunes. I came in skeptical of MC, but it really blew the pants off Wind Waker and gave me hope for Zelda again.
I'm sad to see Capcom go. And it shows, with Phantom Hourglass. Now, Phantom is not a bad game...in many ways it's better than its prequel, Wind Waker; but it doesn't have a sense of fantasy and magic to it. It feels too sterile more often than not, relying on its DS functionality to carry the experience.
OOT was a great game. The only thing I'll say, was that it may have stayed too much in the vein of AlttP in spots. It still managed to make itself its own game, though, and was a blast. Probably has some of the most memorable boss battles. It was missing some of the MM interactive depth, though. I've contended that a successful blend of OOT and MM could result in the best Zelda...
Now, Twilight Princess tried TOO HARD to be the mother of all Zeldas. It took something from every Zelda game and tried to tie it together with what I felt was a disjointed/half-hearted story where Link was basically second fiddle to Midna. TP had good production values, and moments of inventiveness, but if OOT was inspired by AlttP, TP was trying to be OOT 2.0. It had its fun, but it failed to evoke a cohesive Zelda-like spirit. It felt [unintentionally] hodgepodge in design. The Wolf idea was fairly disastrous as far as I'm concerned. It was too distracting, cutting up the quest...at times doing so in a Metroid Fusion sort of way. And, as said earlier, the world was made bigger, but there wasn't an equivalent increase of things to do in it, really. Most times all you could find were rupee prizes that you couldn't even carry because your wallet was maxed out.
(Since combat was brought up, I'd like to relate my thoughts on it: It was great to see Zelda II combat staples return in 3D, but the combat still didn't feel as free form or exciting as Zelda II, which has the best fights in the series IMO). The two-world system, as is, has got to go. It's old hat. At least three other Zeldas have done it, and done it better. (Even Metroid Prime 2 did it--against its own well-being I might add). Let's see a large Zelda world packed with things to do, rather than re-purposed/re-colored alternate universe areas. In the end, it was sad to see this largely uninspired entry, when Shadow of the Colossus was its contemporary.
For my money, Nintendo's last three major entries: Wind Waker, Twilight Princess, and Phantom Hourglass have been very average entries as far as the larger pantheon of Zelda goes. I'm happy to hear that Miyamoto has admitted in a recent interview that even he felt Twilight Princess was lacking in the end, and he hopes this can be corrected in the next entry, which is allegedly a long way off. This next time, I hope it doesn't get over-hyped, and really is an epic Zelda adventure that builds on what has worked before in more significant ways.