Yes. It is not incredibly rare, it's incredibly common. A huge percentage of que... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		brookst 7 months ago \| parent \| context \| favorite \| on: Ask HN: How can ChatGPT serve 700M users when I ca... Yes. It is not incredibly rare, it's incredibly common. A huge percentage of queries to retail LLMs are things like "hello" and "what can you do", with static system prompts that make the total context identical. It's worth maybe a 3% reduction in GPU usage. So call it a half billion dollars a year or so, for a medium to large service.

throwaway2037 7 months ago [–]

    > It's worth maybe a 3% reduction in GPU usage. So call it a half billion dollars a year or so, for a medium to large service.

So if 3% is 500M, then annual spend is ~16.6B. That is medium sized these days?

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact