Deep research, from my experience, will always add lectures.
I'm trying to create a comprehensive list of English standup specials. Seems like a good fit! I've tried numerous times to prompt it "provide a comprehensive list of English standup specials released between 2000 and 2005. The output needs to be a csv of verified specials with the author, release date and special name. I do not want any other lecture or anything else. Providing anything except the csv is considered a failure". Then it creates it's own plan and I go further clarifying to explicitly make sure I don't want lectures...
It goes on to hallucinate a bunch of specials and provide a lecture on "2000 the era of X on standup comedy" (for each year)
I've tried this in 2.5 and 3. Numerous time ranges and prompts. Same result. It gets the famous specials right (usually), hallucinates some info on less famous ones (or makes them up completely) and misses anything more obscure
I tried asking for a list of the most common gameboy color games not compatible with the original dmg gameboy. Chatgpt would over and over list dmg compatible games instead. I asked it to cross reference lists of dmg games to remove them and it ”reasoned” for a long time before it showed what sources it used for cross references, and then gave me the same list again.
It also insisted on including ”Shantae” in the list, which is expensive specifically because it is uncommon. I eventually forbid it from including the game in the list, and that actually worked, but it would continue mentioning it outside the list.
Overselling is only the tip of the iceberg. The real problem is that a lot of managers base their decision to introduce language models into business processes on cutting edge Pro edition demos, but what is, of course, actually used in production is some cheap Nano/Flash/Mini version.
I'm trying to create a comprehensive list of English standup specials. Seems like a good fit! I've tried numerous times to prompt it "provide a comprehensive list of English standup specials released between 2000 and 2005. The output needs to be a csv of verified specials with the author, release date and special name. I do not want any other lecture or anything else. Providing anything except the csv is considered a failure". Then it creates it's own plan and I go further clarifying to explicitly make sure I don't want lectures...
It goes on to hallucinate a bunch of specials and provide a lecture on "2000 the era of X on standup comedy" (for each year)
I've tried this in 2.5 and 3. Numerous time ranges and prompts. Same result. It gets the famous specials right (usually), hallucinates some info on less famous ones (or makes them up completely) and misses anything more obscure