You think that a page that loads jQuery but doesn't use most of it incurs a performance penalty? I'll grant there are a few initializations and dom checks that jQuery does, but it probably blocks for all of 20ms.

...which is significant.

20ms is significant?

Think of it as a fifth of your time budget spent doing nothing.

Most modern browsers are able to paint while loading script files, especially if you put your script files at the bottom of the page, where they belong. So, no, 20ms of parsing, compiling, and a small amount of execution is not a lot of time. Period.