I see A/B testing as just one small component of data-driven decision making.

You can make still great decisions based on your data without doing the thing where you ship the same feature twice and see which one works better.

A priori, data-driven decision making is very powerful. One nice thing about working with former Amazon engineers, is they do bring that mindset. No premature optimization, no belief in facts without data. I think when you get burned a few many times to realize you believe things that are wrong and data shows you to be wrong, you tend to think more about why you know things and what data you have. It's powerful.

By way of clarification, I rarely observed the same feature being shipped twice. It's generally more the 'A' test was the existing functionality and 'B' was whatever change you wanted to make.

The A/B testing platform at Amazon received a lot of investment - it's powerful and so it's used for a lot more than mere A/B testing. It's also a feature toggle platform, lots of things are shipped under that framework so they can be "turned on & off". The metrics collection was powerful and highly integrated. Even if there is no actual comparison in parallel, the metrics that are generated were worth a lot (and often needed, as everyone's goals are all data based, so you want data to show you are hitting those goals. Can't just say we launched 5 features, and can't say we are just still here, it's data or nothing at Amazon)

Most importantly, the "great decisions" is interesting. The power of gathering data very quickly and often is you see when decisions were not great. Amazon was excellent at this, things that made more money were given support, things that lost money were quickly terminated and re-organized. The data provides light on the decisions, it's virtually a BizDev super-power to be able to pivot away from losers so quickly like that.