This was interesting and sent me down a research hole.
General conclusion:
Corporate litigation is mostly just a series of self-investigations so that both sides can learn what both sides actually know, given that neither side knows much about themselves OR the other side. At the same time both sides are trying to stop the other side from getting the judge to order them to do more investigating.
Humans must not anthropomorphise {non-humans}
Humans must not blindly trust the output of {anything}
Humans must remain fully responsible and accountable for consequences arising from the use of {anything}
Naturally, none of this advice matters at all as humans will do what they do. This just documents a subset of the ways real humans consistently make choices to their own detriment.
I kind of agree with 1, but not really with 2 and 3. It's easy to come up with trivial examples where it is both unreasonable and not feasible to follow those two, both for AI and non-AI scenarios.
From the corporate POV, the lesson remains: never ever ever conduct research that could lead to a conclusion that your product/service can harm in any way, unless you know how and intend to fix/change it.
The most important evidence was just internal research saying exactly what the plaintiffs wanted.
> We have incorporated these findings into our recent evaluation efforts. In the last months we’ve chosen to report results from the public split of SWE-Bench Pro. We recommend other model developers do the same. SWE-bench Pro is not perfect, but empirically seems to suffer less from contamination issues.
Additionally, one can argue that the state of the world has changed enough that assumptions made by the USC at the time of precedence require reversal.
General conclusion:
Corporate litigation is mostly just a series of self-investigations so that both sides can learn what both sides actually know, given that neither side knows much about themselves OR the other side. At the same time both sides are trying to stop the other side from getting the judge to order them to do more investigating.
reply