1v1 Villagers (especially Berserkers) are crazy tough

But they have lots of big weaknesses.

1. Movement being the main one - no recall, hamstring, summon traps or just being able to run generally from fights is tougher

2. No healers - plague, poison, bleeding, etc ruins them (unless defender or near village)

3. Take the Head - they're fodder

4. Finally, you don't need to fight them. As a non mage they shouldn't hunt you and as a mage they should be fairly easy to avoid (point 1)

RBWs get lots of kills but their ratios are normally fairly low too.