If you write your own library to do it, there are strictly more things that can go wrong because at some point you're implementing a HMAC validation thingy. If you use a third party library, you're exposed to bugs that stem from JWT's design flaws even if you only allow a specific key, like the RS256 v HS256 bug I mentioned.
You are definitely right that if for some reason you must do JWT, the way to do that is to strip as much of it away as possible. In particular, if you wanted to do safe HS256-only, you'd ignore the header, decode the body and tag, and validate the tag.
To be clear, I didn't write a cryptography library... I wrote a JWT wrapper for validating JWT for the system in question. This was also before there were JWT libraries for a number of platforms. Beyond all of this, the JWT validation worked across a number of tokens generated from an alien (to the app in question) application, and rejected when it didn't validate.
Also, it was literally only validating the tag and ignoring the header... it was using an asymmetric key for signing
You are definitely right that if for some reason you must do JWT, the way to do that is to strip as much of it away as possible. In particular, if you wanted to do safe HS256-only, you'd ignore the header, decode the body and tag, and validate the tag.