It was very distracting. It takes quite a bit of training to eliminate pauses from speech but it takes significantly less effort to replace 'um' with a pause.

Pauses aren't even bad. Pauses give your audience an opportunity to process what you're saying. I've had pauses encouraged to me.

The problem is, like the FedEx arrow, once you see (hear) it, you can't avoid doing so in the future.

It is an error to assume people who pay attention to trees don't understand forests.

I think that many people don't consciously note a thing, but it still has an effect upon them.

This does not apply to PG, but sometimes I find myself disliking a speaker but not quite realizing why until hours after the discussion when I can freely introspect and disentangle my emotions.

Simply understanding your own biases, and emotional peeves doesn't make you elitist which is what you seem to be implying by mentioning the "Toastmaster types".

It distracts, no matter if you count or not. And really, if you do a lot of public speaking and respect your audience—do a favor to your audience and try to improve. Sure, audience will bear with almost anything and more so if what you say is interesting—but why make them suffer?