1. Write up a detailed spec as a prompt, including a bunch of detail about the actual functionality. Add in some details about wanting to test everything. At this point I did not specifically say "Hey LLM, in my tests, do not mock one of the key idempotency functions in this system". This is obvious to me since 1/4 of the system was concerned with idempotency, and this was super clear in my original spec
2. LLM works for 30 minutes, spits out 2.5K lines
3. I review the product functionality and the test cases at a glance, and then start running tests and iterating on the software.
If I have to mentally split every line of the LLM output and decide whether it's correct or not (remember, I didn't write the rest of the system), then I lose all the productivity gains that the LLM offers. You don't do this, no one does this unless they're modifying a few lines at a time.
My point is that you cannot build robust software from scratch with LLMs right now. They walk right into footguns we've spent decades learning about and building solutions for.
1. Write up a detailed spec as a prompt, including a bunch of detail about the actual functionality. Add in some details about wanting to test everything. At this point I did not specifically say "Hey LLM, in my tests, do not mock one of the key idempotency functions in this system". This is obvious to me since 1/4 of the system was concerned with idempotency, and this was super clear in my original spec
2. LLM works for 30 minutes, spits out 2.5K lines
3. I review the product functionality and the test cases at a glance, and then start running tests and iterating on the software.
If I have to mentally split every line of the LLM output and decide whether it's correct or not (remember, I didn't write the rest of the system), then I lose all the productivity gains that the LLM offers. You don't do this, no one does this unless they're modifying a few lines at a time.
My point is that you cannot build robust software from scratch with LLMs right now. They walk right into footguns we've spent decades learning about and building solutions for.