How come it is so hard to call someone out for their pive aggression?

Go ahead and ostracize him. The Spanish have a saying."Take what you want and pay for it, says God" If he chooses to behave that way, let him take the consequences. He's just a jerk.