It kind of depends on what the use case is. If it's simply to query against a large amount of information, you can just create embeddings of the information in chunks and add these together in a vector store index (https://gpt-index.readthedocs.io/en/latest/guides/index_guide.html). Then you embed your query using the same model and basically the relevant chunks are returned, and then you can synthesize a response from those chunks.

So let's say the use-case is to create a conversational tutor assistant for a textbook. Obviously, you can't put the whole textbook in the prompt. So you feed it in one paragraph at a time into the embeddings model, and store all these embeddings (along with the text they relate to) in a vector database like weaviate or pinecone. Then, when the user asks a question, you embed the query using the same embeddings model, and do a cosine similarity search using your vector database (a common function of vector DBs). And you say, return me the top 5 relevant chunks. Now you have some short context you can feed into normal GPT-3, with a prompt like "given the following context, create a bullet point summary" or "given the following context, create a simplified analogy using real-world examples."

Embeddings are basically the first half of the transformer. Language transformers essentially have two halves - the first half understands the input and encodes it into a set of numbers the model can understand. The second half takes that understanding and predicts a probable next word. When you think about this from a computational perspective, the first half only runs once, and the second half runs hundreds of times (once per output token). So you end up with only a fraction of a percent of the computation time spent on understanding (embedding) the input, and most of the time iteratively generating tokens. What semantic search in vector space lets you do is essentially compare items after only step 1, and THEN produce an output once you've gathered the necessary context. But of course you perform the embedding on your data ahead of time, so the only real compute that is needed at runtime is the embedding of the user's query, which is cheap.

I believe so, I think you can experiment with different summarization prompts too. For me it is still trial and error when dealing with large context windows.