In terms of "realism" (yes, I know we're talking about a fantasy game with elves and giants), I'm fine with elves not getting hit as often. Fly like a butterfly and all. Giants are large and slow moving. However, being so large, they should have like 3x the hit points of an elf.
That said, they should hit *other people* less often too. Because, again, they're big and slow. But when they hit you, it should do like 3x the damage. Every hit should be the equivalent of a deathblow.
Mechanics wise that would suck, though.