If something could be very harmful you should avoid it out of precaution. And many of these chemicals are known endocrine disrupters, so just because there is no slam dunk evidence there is a lot of circumstantial and anecdotal evidence that indicates a problem with these substances.

Plus, the people who avoid them are almost invariably healthier, so taking steps to avoid them seems to have positive effects anyways.