Great question. More than likely everyone has a different answer, but each person, group, nation etc... has some implicit goals. I would argue the current human goal is "reproduce successfully" as that is what is baked into our genes - I doubt that is actually optimal though.
The failed idea of a coherent extrapolated volition (CEV) that came up years ago was (roughly) the idea of using revealed preferences to understand what Humanity's goal is. This would give us a benchmark for what an AGI's goal should be.
So if you want to be able to measure the capability of an AGI in comparison to human system, you need to understand the set of goals in humanity and then compare them to the AGI outcomes.
The concept of goal direction in AGI is hugely contentious - but make no mistake there needs to be a goal if it is going to actually function at superhuman levels.
The failed idea of a coherent extrapolated volition (CEV) that came up years ago was (roughly) the idea of using revealed preferences to understand what Humanity's goal is. This would give us a benchmark for what an AGI's goal should be.
So if you want to be able to measure the capability of an AGI in comparison to human system, you need to understand the set of goals in humanity and then compare them to the AGI outcomes.
The concept of goal direction in AGI is hugely contentious - but make no mistake there needs to be a goal if it is going to actually function at superhuman levels.