This is dogmatic reasoning. If censorship wasn't necessary, neither would the government. At sufficient scale, humans stop behaving in the best interest of the group, and you need ways to correct that. It'll never be perfect, but much like democracy, we don't know of a better system.

Surely, in a conversation, the most damaging thing you can do to the integrity of that conversation is to selectively nullify the voice of a participant.

It ceases to be a conversation then. It is something else, posing as a conversation.

Maybe it would be better if this censorship-power is democratically controlled. But if this power is given to an individual. Well that's different.

In order to have a conversation at all, you need to deal with the guy with the megaphone first. If you do not have moderation, you drown in spam, and do not have a conversation at all.