Framing it differently: I have $100,000 as a down payment for a $500k house. At 8.05% the monthly payment is $3,686. At 3.11%, it's $2,138. That's a savings of $1,548 monthly. 12 payments a year for 30 years that's a net change of $557,280.
You’re not considering that the market responds to these changes in affordability. If you can afford $3,186/mo today and the interest rate plummets, sticker prices will (over time, generally) track higher such that your monthly payment remains roughly the same.
It’s not a perfect relationship between the two—affordability does fluctuate, and the movements have a lot of latency between them—but on a societal scale the relationship holds.