To accept giants in the Bible means people would have to accept other parts of the Bible many churches refuse to acknowledge as true scripture. The whole story around the giants is heresy in many sects.
It's also possible God, the angels, and even giants were all just aliens who visited Earth long ago. I mean, Cthulhu has big feet, too.