You can’t include “all” the context due to the prompt token length limitations. You have to use techniques for surfacing the right context that would fit in the limited context window and sending that along. There are techniques that keep a running and continuously compressed summary, and techniques that use embeddings to hone in on relevant chunks of past conversation and context and send that along based on some ranking and the size limitations.
Right I am familiar with that stuff I was just trying to answer for the case that useful context fit within the max_tokens. Your startup looks really interesting. Kind of similar to some of the plans I have for aidev.codes. It seems for code you would need to just send a whole file usually.. maybe a whole function? Because if you break it up too much its doesn't necessarily "compute". I dunno. Are you using code-davinci-002 from OpenAI, and if so, have you managed to get your rate limit increased? They seem to ignore my support requests. text-davinci-003 is pretty good too.
One experiment I have done for getting context is just to say "given this user update request, and this directory listing, which files would you need to inspect" then the next prompt includes those files and asks to modify them to complete the request, with a specific format for identifying filenames in the output.
Exactly! This is also what I’ll be implementing within the next few days. Embedding conversation history and dynamically doing semantic search for relevant conversation snippets and building a prompt based on that to send to GPT3
