You're not crazy, lots of people agree with you. Just please don't be like so many SF residents and complain that all the people "who made SF what it is" are being replaced by tech hipsters making 6 figs, while simultaneously complaining about any new construction in their neighborhoods. It's a hypocritical position. There either needs to be more tall buildings or people with sub 90'th percentile incomes are going to have to GTFO from SF until Silicon Valley collapses.
Transportation. The bulk of the economy is still on the west side of the Bay, and crossing the Bay can be a huge pain in the ass. BART is frequently unreliable, expensive, infrequent, and stops early.
On top of that the transportation on the Oakland side is also highly lacking, further increasing the friction for people working in SF but living in the East Bay.
On top of that the transportation on the SF side is also highly lacking. If you work within walking distance of a BART stop, power to you. For everyone else it means catching a transfer to the worst transit agency I've ever seen: MUNI.
In other words, the only commute that really competes with living in the city proper is one where you're living walking distance to a BART stop, and you work within walking distance to a BART stop at the other end. Any other use case becomes an irredeemable mess, substantially limiting the attractiveness for residents.
Transportation has, and is, and will continue to be, the Achille's Heel of San Francisco and its surroundings. The continuing utter failure to invest in transit infrastructure will be the city's undoing.
On the bright side, this is great for Oakland.