It’s definitely not. Our prompts that were generating JSON output went from around 95% valid JSON to about 10% overnight. The model just started inserting random commentary. We’ve reverted to the 0314 model and it’s working fine again.
I use ChatGPT (GPT4) to build scaffolding for python one-off scripts, and over the past 3-4 days I'm getting nonsense. Not python-looking nonsense, but markdown, weird quotes, random text, etc. Same prompts.
I had an API integration written to convert an English language security rule into an XML object designed to instruct a remote machine how to comply with the rule programmatically. April 2023 we had about an 86% accept rate, that number has declined to 31% with no changes to the prompt.
This is the kind of info I've been looking for - I ran some informal experiments which asked ChatGPT to mark essays along various criteria analyzed how consistent the marking was. This was several months ago, GPT-4 performed quite well, but the data wasn't kept, (it was just an ad-hoc application test written in jupyter notebooks).
I'm certain it's now doing significantly worse on the same tests, but alas I have lost the historical data to prove it.
Have you tried using the recently releases function calling API? That’s reliable at returning json in my experience, although I’ve just tinkered with it, not used it for anything “real.”
My guess is that the degradation of JSON capability happened recently? The gpt-4 API switched over to gpt-4-0613 (the function calling version) on June 27. And given the performance increase for ChatGPT Plus at the end of May, my guess is they started testing the new model (which is much faster) on web users around then. In my testing [1], the new version is:
a. Worse at general code-like tasks without using functions
b. Equivalent or better at code-like tasks if you use the function API
c. Much faster than the older model either way.
I'd guess it's cheaper to run, too, and that they use the presence of a function in the API signature to weight their mixture of experts differently (and cull some experts?). The degradation in general purpose coding tasks is pretty obvious and repeatable (try the same prompts in the Playground with the -0314 model vs the -0613!), but it does seem like you can regain that lost capability with the new function call API, and it's faster. The tradeoff is that you only regain the capability when it calls functions; you can't really have a mix of prose-and-code in the same response as easily, or at least not with the same quality.
