I've had a similar experience, although my learning was "people, even experts, are really terrible at understanding which treatment will perform better".

We always run >=3 variants, surveyed the dozen team members on which one they thought would run. Over the years, there was no clear pattern over who could make that prediction.

IE, it's not possible to predict which is the most effective treatment, even when you include a really bad idea in the treatments!