The data suggests your feelings are incorrect, and that things were not as dangerous as you feel they are. I'm not even sure what you mean in this case at all: people still washed their clothes, garden tools are not any more sterile now than they were then, and refrigeration can be worked around by different dietary and food storage choices. (Ironically the need for refrigeration partially came with societal advancement: fresh milk for example was traveling farther and farther and increasingly transactional circumstances, which meant it was more likely to carry food-borne diseases whereas, when you go further back, nobody was waiting days to drink their fresh milk and thus didn't worry about it.)

I’ve no doubt that there was a statistical impact, but it is probably rendered invisible by such things as skyrocketing obesity, heart disease, and cancer rates caused by the modern environment and lifestyle. This is probably especially true when you consider the QoL of many older people today. This is also apparent when the cited article mentions that many fewer people died of heart disease, but they did die of heart damage caused by infections. They lived just as long, you might think antibiotics would keep us going even longer, but we tread water because now we simply die of heart disease at about the same age or a little younger instead.