Did something similar recently as well. I did not notice any need for plugins from day one of switching. Fzf can replace a lot of plugins if you know how to use it.

Things like git branch name and virtual envs are handled by starship.

My custom config is less than 10 lines now:

  export HISTSIZE=1000000000
  export SAVEHIST=$HISTSIZE
  setopt EXTENDED_HISTORY
  setopt autocd
  autoload -U compinit; compinit
  source <(fzf --zsh)
  eval "$(starship init zsh)"
  set -o vi