"others still might" is a cheap way to disregard any ethical consideration.

I can always make up a hypothetical scenario where I become the victim of some aggression... to justify being a bully.

Even if this is your motivation (we will never know for sure, right?), it could still decrease probability of you being the victim of said aggression.