Has anyone measured the performance of very large context windows like this vs a good RAG that you also constantly update and curate?
At least with other very large context windows like for example Claude offers a RAG is still very much preferable as it avoids confusion and collisions with information in the context that isn’t correct or relevant.
Sure you can also prune the context window and for many existing models you also need to do that (I often use an LLM to summarize a context to keep it going) but doing it with a RAG seems to still be much easier. This especially holds true of you use good knowledge management techniques to structure your RAG so your retrievals are optimized.
P.S. on a side note how confident are we that these very large context window models are not just a RAG in disguise? As the models which boast very large windows are at least for now all locked behind API access only.
At least with other very large context windows like for example Claude offers a RAG is still very much preferable as it avoids confusion and collisions with information in the context that isn’t correct or relevant.
Sure you can also prune the context window and for many existing models you also need to do that (I often use an LLM to summarize a context to keep it going) but doing it with a RAG seems to still be much easier. This especially holds true of you use good knowledge management techniques to structure your RAG so your retrievals are optimized.
P.S. on a side note how confident are we that these very large context window models are not just a RAG in disguise? As the models which boast very large windows are at least for now all locked behind API access only.