Yeah he was frustrated, it's a very powerful item (for his build specifically) which you couldn't personally make use of. However I don't think anyone would call you a douche for keeping it, I wouldn't want my foes having such powerful items either.
IMO on non-Magi, if they've declared you an enemy and you know they're going attack then you can strike whenever. Often tactics would dictate that you should strike first.
The only time you would get in trouble is if you're targeting, hunting and prioritising these non-magi enemies over magi.