In theory yes, but in the most populous states we also have large stamp duty taxes that get charged every time you buy a house that lock people in. For a $1 million house I bought this came out to ~$40k. In general people usually renovate their houses or stay inside houses that are way too big for them instead of moving into one that better fits them.
What's mental is that people buying $1m houses aren't anywhere near rich. That's just how much houses cost now. You spend years saving, and then the Government pinches most of your deposit!