> Looking at the kind of mistakes Claude made, it seemed like a human could make those mistakes too (very useful feedback for us to improve our API ).
This is something we've found for our API -- just having LLMs attempt to use it helps us identify things that we haven't documented well or placed enough emphasis on (for things that are critical but are non-obvious or may be drowned out by other less important information). Improvements that help the LLM tend to be good for developers too.
Life imitates art: afar you’re describing there is basically _The Secret_ (I.e. if I wish hard enough for something then eventually it will come true), except it’s LLMs that get wish-fulfilment, not us.
Huh, you've reframed The Secret for me - now I see it being, if you wish hard enough for something then eventually the Universe will make it so, just to shut you up.
This is something we've found for our API -- just having LLMs attempt to use it helps us identify things that we haven't documented well or placed enough emphasis on (for things that are critical but are non-obvious or may be drowned out by other less important information). Improvements that help the LLM tend to be good for developers too.