My biggest issue with A/B testing isn't even mentioned here... gaslighting your customers is absolutely not OK. Particularly with older people, the constant "where the f..k did Outlook now put feature XYZ" (in the case that comes to my mind, the CC bar which used to be tab-reachable, now you have to tab+space or manually click on the tiny gray "cc" letters) onslaught is just absurd. When you change how applications behave without telling the users, it's a direct attack on their muscle memory at best and makes them question their sanity at worst.
My second biggest issue is: it's rare that companies offer actual, live-human support these days anyway. When marketing adds A/B testing, shit becomes really annoying if something breaks as a result - usually the phone lines are suddenly flooded, the agents have no idea what has happened either and try to reproduce and figure out what's going on (and sometimes can't because they aren't part of the test group!), and so even people who haven't been in the testing group are going to be very pissed off.
IMHO, A/B testing without explicitly notifying the customers in advance should be banned by law, and that ban be harshly enforced. Customers are not guinea pigs, and with the rise of elderly people on the Internet this becomes an actual public safety issue (as ever-changing stuff makes it easier for scammers!).
You're downvoted, but this issue is more common than it seems; and I agree, more serious than it seems.
You're describing adversarial UI changes to small populations of then unsupported customers. This can have outsized impacts on vulnerable populations, eg., esp. the elderly.
Not to mention the lovely "This option that used to be here no longer is here." getting the response of "its still here (for me)". Youtube specifically loves to do this to me.
> Outlook now put feature XYZ" (in the case that comes to my mind, the CC bar which used to be tab-reachable, now you have to tab+space or manually click on the tiny gray "cc" letters)
I can sort of understand wanting to hide stuff on mobile, but the discovery of controls to unhide things should be better. I often help people that are stuck trying to figure out how to do something in an app and not realizing they can click on something that gives no indication it's clickable is a common thing.
Desktop is another world. I often have 20+ inches of horizontal space and a hamburger menu. It's infuriating, especially when the hamburger menu is hiding one button.
As Product manager/owner I've only found A/B testing useful when trying to narrow in on a specific demographic and you are trying to find some optimization.
The marketing/sales funnel use of it is kind of gross and has ruined , imo, something that has utility in a very narrow scope.
Cheers, also very much agree customers should be informed and allowed to opt out.
'hey we have a new UX to try..would you like to switch?' the data from people that opt-in is way better
My second biggest issue is: it's rare that companies offer actual, live-human support these days anyway. When marketing adds A/B testing, shit becomes really annoying if something breaks as a result - usually the phone lines are suddenly flooded, the agents have no idea what has happened either and try to reproduce and figure out what's going on (and sometimes can't because they aren't part of the test group!), and so even people who haven't been in the testing group are going to be very pissed off.
IMHO, A/B testing without explicitly notifying the customers in advance should be banned by law, and that ban be harshly enforced. Customers are not guinea pigs, and with the rise of elderly people on the Internet this becomes an actual public safety issue (as ever-changing stuff makes it easier for scammers!).