The people who consume the notarized documents. If too much crap comes through they can reject the issuer. Kind of like how Symantec CA got dropped by browser makers.
Public notaries are licensed by US state governments. There is generally a background check, brief training course, and application fee. In at least some states they have strict liability for theft of their stamp.
What does it mean to reject the issuer when there are around 4.4 million notaries in the US? What systems are in place now or would need to be created in order to aggregate trust and what are the pros and cons associated with those systems?
For individual notaries file a complaint about incompetence or report them for fraud. Signatures, seals, and watermarks aren't as good as public crypto but that's okay because phone calls, clearinghouses, and the legal system backstops them (especially for reversible transactions).
Rejecting issuers would be more applicable to repeated transactions from a corporate certificate authority.