I liked PoR and thought it was definitely one of the more fun Castlevania games in the Metroidvania genre. The dual character system is a lot of fun too because Johnathan and Charlotte compliment each other well. They're both very useful (Charlotte probably moreso IMO) and I found myself using them both pretty equally once I got the hang of the game.
I think the biggest problem with PoR is the wasted potential. The level design isn't great, the dual character system could have really been made much deeper than it was, and had the art been a little more dark, the story could have been a little more believable.
That said, it was meant to be an action game, so I can forgive the story being slightly lame. After all, LoD and LoI are about the only CV games with a good story that I can see.
There's plenty of replay value as well and there's so much gear you can get than you could conceivably never play the game the same way twice.
As for the remixed portraits, everyone lauds SoTN for the Inverted Castle, and the remixed portraits are a good bit different from the first ones as opposed to the Inverted Castle which is exactly the same thing but upside down, so I don't get why that's a sticking point.
In the end, it doesn't MATTER why people don't like it. It's not like we're all paid reviewers (and even if we were, it STILL wouldn't matter.) If you like it, just enjoy it for what it is rather than picking it apart. It's a solid action game that has some flaws, but it's nowhere near as bad as everyone makes it out to be.