I realized if someone were to assign me the ticket for fixing this behavior, I would have no idea where to begin with solving it even with this blog post explaining the problem, so I'm very curious to know what the most practical solution is. (They obviously aren't adding "If someone asks you about a seahorse emoji, there isn't one available yet, no matter how strongly you believe one exists." to the system prompt.)
I bet they probably are adding that to the system prompt at least in the short term while people are paying attention before looking for a longer term answer.
The system prompts I've seen are absolutely massive.
> This attention scarcity stems from architectural constraints of LLMs. LLMs are based on the transformer architecture, which enables every token to attend to every other token across the entire context. This results in n² pairwise relationships for n tokens.
The n² time complexity smells like it could be reduced by algorithm engineering. Maybe doing a preprocessing pass to filter out attending to tokens (not sure what the right term of art is here) that do not contribute significantly to the meaning of the input. Basically some sort of context compression mechanism.
People really really want LLMs to output a highly reliable finished product, and I suspect we're probably never gonna get there. Lots of progress over the past couple years, but not on that.
I think it's much more interesting to focus on use cases which don't require that, where gen AI is an intermediate step, a creator of input (whether for humans or for other programs).
I agree, but I still suspect OpenAI and other LLM companies do stuff like that, when an example of a hallucination becomes popular.
If I see some example of an LLM saying dumb stuff here, I know it's going to be fixed quickly. If I encounter an example myself and refuse to share it, it may be fixed with a model upgrade in a few years. Or it may still exist.
Something about how you have to keep repeating "There is no seahorse emoji" or something similar reminded me of the Local 58 horror web series where it seems like the program is trying to get you to repeat "There are no faces" while showing the viewer faces: https://www.youtube.com/watch?v=NZ-vBhGk9F4&t=221
"This behavior is a function of the core AI technology we use, we are unable to resolve this issue with a standard software patch or update at this time.
For the time being this issue can be mitigated by not asking about seahorse emoji.
We are closing this support ticket as the issue is an inherent limitation of the underlying technology and not a bug in our specific implementation."
