While at some level that’s true (autocomplete) at another level it unlocks a abductive reasoning ability for machines that prior AI failed at. While it’s not reasoning per se, it absolutely makes probabilistic inference over an abstract semantic space that’s remarkable. For instance you can use a multimodal generative AI to take a photograph of a saloon and ask it how to make money and it’ll describe playing poker at the poker table and working at the bar for money, then when prompted describe how to navigate to the table given the objects in the room. This is a remarkable extension to current AI - which could actually perform the navigation and plan routes, even use goal based agents to instruct the generative model to plan a way to make money. I actually am not that worried about the risks of wandering mind or hallucinations, I’ve found ways to detect when it wanders (for instance, creating an api to call including an echo() for textual response and validating the output and regenerating responses until it conforms, then doing domain verification on the API parameters)

But that gets to one of my core beliefs - LLM and other generative models are tools who require constraint enforcement, agents to direct, verification and validation, deference to inductive and deductive systems, optimizers, solvers, etc. The fact they can’t compute primes or solve quadratic equations doesn’t impress me - because we have those tools already. Focusing on what they’re weak at and ignoring what they’re amazing at is foolish and really small minded. It’s interesting that you can train them to do some of these tasks, but trying to use them as a calculator when we have calculators is absurd, trying to use them as information retrieval systems is doomed to fail, trying to use them to be a complete solution to literally anything is simplistic.