It seems obvious to me that we just don't mind delegating this task some of the time, as long as we largely agree with the results.

I don't think you need an appeal to authoritarianism to appreciate a forum that isn't 80% penis-enlargment ads. A few people spending their time to moderate this content saves the whole group a lot of time.

In the same way that I'm okay with other people deciding a tomato is too moldy to sell (whether that's the farmer, health inspector, or grocer), I'm also okay with some people having the power to remove the equivalent speech from certain spaces.

You just need to be more careful when the jurisdiction becomes larger, because it becomes harder to "vote with your feet" to go to a place whose policies you agree with.