Yeah we got dinged by our pentesters a few years ago because the LB didn't clear X-Forwarded-For headers. So you could just set some trusted IP into the X-Forwarded-For header and various ip whitelists went "Well, it came from there, so we gonna let it though".

Oops :)

It is one of these trust-based headers that need to be cleared at the edge of your network / trust zone.