We're excited to announce our state-of-the-art reranker, the first that can follow custom instructions about how to rank retrievals and prioritize information. Based on BEIR and our internal benchmarks, it's also the most accurate reranker in the world—even without instructions.
RAG systems frequently have to resolve conflicting information in knowledge bases. Marketing materials can conflict with product materials, documents in Google Drive could conflict with those in Microsoft Office, Q2 notes conflict with Q1 notes, and so on. Traditional rerankers only sort retrieved documents by relevance to the query, so we wanted to build something that could resolve these conflicts by following your instructions about how to prioritize retrievals beyond semantic similarity:
“Prioritize recent documents over older ones”
“Prefer PDFs to other sources”
“Give more weight to internal-only documents”
The reranker also achieves state-of-the-art results on the industry-standard BEIR benchmark, along with outperforming other rerankers on real-world customer datasets.
Getting started
The reranker is available today as a seamless drop-in replacement for your existing reranker (or as an easy addition if you don't use one yet).
The first 50M tokens are free. Just create a Contextual AI account (https://app.contextual.ai/?signup=1), visit the Getting Started tab, and use the /rerank standalone API.
I’m Ishan, Product Manager at Contextual AI.
We're excited to announce our state-of-the-art reranker, the first that can follow custom instructions about how to rank retrievals and prioritize information. Based on BEIR and our internal benchmarks, it's also the most accurate reranker in the world—even without instructions.
RAG systems frequently have to resolve conflicting information in knowledge bases. Marketing materials can conflict with product materials, documents in Google Drive could conflict with those in Microsoft Office, Q2 notes conflict with Q1 notes, and so on. Traditional rerankers only sort retrieved documents by relevance to the query, so we wanted to build something that could resolve these conflicts by following your instructions about how to prioritize retrievals beyond semantic similarity: “Prioritize recent documents over older ones” “Prefer PDFs to other sources” “Give more weight to internal-only documents” The reranker also achieves state-of-the-art results on the industry-standard BEIR benchmark, along with outperforming other rerankers on real-world customer datasets.
Getting started The reranker is available today as a seamless drop-in replacement for your existing reranker (or as an easy addition if you don't use one yet). The first 50M tokens are free. Just create a Contextual AI account (https://app.contextual.ai/?signup=1), visit the Getting Started tab, and use the /rerank standalone API.
Documentation: /rerank API docs: https://docs.contextual.ai/reference/rerank_rerank_post Python SDK: https://github.com/ContextualAI/contextual-client-python/blo... Langchain package: https://pypi.org/project/langchain-contextual/
Happy to answer any questions about how our reranker works or how you might integrate it into your RAG systems!