This is especially common with a fresh USB port. I learned why recently from the book Open Circuits: The Inner Beauty of Electronic Components. There is a mechanical component that is more stiff before being used, which can lead one to think that they simply had the USB connector the wrong way around.
I highly highly recommend the book by the way, to anyone on HN. Goes all the way from mechanical components to resistors, nixie tubes, CMOS sensors, processors etc. Excellent photography that reveals the internals, along with operation explanations and history for each component.
Fun fact: part of the licensing agreement to use USB, is to have the usb symbol on top of the connector. So unless you're using a cheap unlicensed cable, look for the symbol facing up and you'll always be correct
After all this time I'm not going to trust a decoration.
When I'm connecting an USB cable horizontally I always think "hol up". So I always remember to plug it with the hole up.
When I'm connecting it vertically I think "hold right". So I know that if I want to hold it right, I have to put the hole right.
Has worked for me so far.
EDIT: Though "left" and "right" are more vague compared to "up" and "down", so connecting vertically assumes a specific "mental point of view". I'm just using one that's intuitive for me, but it might not work for everyone.
People think the USB-A port looks symmetrical. The way I do it is I look inside the connector and note which side the pins are before trying to plug it in 3 times.