That is the main worry around the topic. The world is full of principal–agent problems. Politicians are supposed to serve the voters but they seem to be co-opted by their own self-interests and they go against the interests of the citizens. Burocracies and charitable organisations spend their energy to enlarge and entranche themselves instead of solving the problems they were created to solve for. CEOs optimise for their renumeration package instead of the long term benefits of the shareholders. Tradespeople cut corners where they know they won’t be caught before their payment is due. Police forces becomes corrupt and prefers to play statistical reporting games to look good instead of fighting crime.

Our track record of aligning inteligences to goals is abysmal. We have a few tricks and we try them, but they are not that great. If we extrapolate this to artificial inteligences it is not difficult to foresee great catastropes.