Is that irony, though? Not that I downvoted the parent comment or anything, but inasmuch as downvoting is saying "I don't think your comment is of high quality", that's totally not inconsistent with free speech.

There's a difference between saying "I think XYZ is wrong" and saying "I don't think people should be able to say XYZ", and downvoting feels much more like the former.