I was an OMZ user pretty much since I adopted ZSH many years ago. I never really experienced much of the slowdown(s) that people tend to complain about. Even still, getting my dotfiles set up on a new machine was always a little bit of pain. I'd try to install OMZ from my OS' package manager, remember that that rarely worked perfectly, gone back to OMZ's webpage and copied the cURL command... the fought with FZF shell integration for about an hour. Finally one Saturday I just stripped it all out. I went with raw ZSH and Antigen for the few plugins I care about. Spinning up new machine is now painless. And I always like simplifying things.
Thing is, a lot of how OMZ affects your zsh usage is based on usage patterns. For instance, I usually have zsh running in a terminal that is always open and running. Now, if I close my terminal and restart, then I notice that OMZ is kinda slow and it irritates me, even if it's only for a second. But that almost never happens.
Anyway, I'm going to be revisiting my zsh config soon and I'll be looking for ways to make sure I'm only using plugins that I absolutely do need. My principle in the past with workflows in zsh and Neovim is that I only add crucial plugins and try to keep my config minimal. Maybe I've gotten away from that a bit and need to regroup, so I find threads like this one useful to keep up with what's current.