They key to a creepy or scary atmosphere is to leave the player feeling like they are ALMOST in control of the situation. Almost, but not quite. Dead Space 1, 2, and iOS all managed this quite well, giving you all the tools you need to manage yourself and even do very well in combat, but still feel like things are a touch beyond your control. Yes, you're a badass, but you're constantly feeling like it won't be enough.
Compare this to Castlevania. In most games in the series, eventually you're screaming "BRING IT!!" instead of "ohshitohshitohshit", and often that happens very early on in the game, when you lose your scripted helplessness and get all the tools you really need and better weapons than you start with. Lords of Shadow did a lot well, but it strayed the furthest from those creepy horror vibes because you don't even start with scripted helplessness. Gabriel, Simon, and Trevorcard are straight up badasses from moment one; shredding werewolves and other beasties of the night with aplomb doesn't engender a creepy atmosphere. That initial fight in Lords of Shadow 2 before Zobek rescues you is easily the most tense fight of the TRILOGY because Dracula is actually helpless, and that's the feeling a creepy or scary game needs to have ALL THE TIME. No matter how strong you are, you feel just a little bit helpless, like you're out of your league, or that there's something more that you could have done earlier to prepare but backtracking is impossible and you're stuck with the decisions you've made, be they good or bad.
The other half is good art direction and great sound design. Bloodborne was a master at all of this. It had, in my opinion, the ur-example of good creepy art design for a Castlevania game. All that Victorian London Gothic with a heavy sense of undead curses and infinite urban decay cranked up to levels that shouldn't even be possible made for a spooky as hell atmosphere. The relative silence as you navigated the city of Yharnam pounded home a sense of loneliness and isolation that isn't easily duplicated, but should be. The silence was punctuated only by the sounds of the demented citizens and the horrible monsters lurking around the next corner, and worked to build up dread in the player, while the combat music was like the snapping of a rubber band of tension. Visiting the Hunter's Respite felt like a breath of fresh air every time, but no matter how much time you spent in that sanctuary, it didn't save you. You'd have to go back, and you knew it, and you dreaded it.
But Bloodborne wasn't a horror game. It had all the trappings, but it's not a horror game. It's an open world action game, just one with intimidating foes and killer atmosphere.
Were I to make a Castlevania that returned to the themes of horror and fear, that's where I'd start my research.