To create an AI that people will want to use, we need people to use it so we get better data, improve it, and learn how to control models. It’s a catch-22. I definitely don’t want AI tabs in Firefox but I can see why all this might be necessary because of massive potential for a superintelligence to solve problems or open new doors we didn’t know existed. But then we have to ask ourselves if these new pathways for humanity are so far removed from what humans can accomplish on their own, is there pride in a new revolution akin to the human derived pride in what we have accomplished on our own for the Industrial Revolution, for example, or even just petrol or nuclear power discovery?