It will definitely leach from PVC pipe. I'm not sure about PEX. Yes to water dispensers in refrigerators.
In general, the problem will be worse with smaller diameter pipes/tubes due to surface area/volume ratio, worse if the water has been standing in the pipe/tube, and worse if the water is hot.
These chemicals are extremely difficult to get away from. I saw a study where they tried to get volunteers to take a reasonably large set of actions to reduce their levels of BPA and Phthalates, and they were able to get them down to about half the original levels, but not lower.