Both are important. A lot of poverty is directly caused by having a class of rich people who seek to siphon away as many resources as they can. A society with only poor people would probably be a more cooperative one, which would help in raising the average standard of living.

For instance, shelter being expensive implies either that it's fundamentally difficult to construct shelter (which is not the case, as seen in every slum and homeless camp everywhere) or the existence of rich landlords.