Modern dense retrieval systems rank primarily by semantic similarity. They find text about the same broad topic, but they know almost nothing about why the question is being asked, what task the user is in, or which constraints make a result useful.

As in any research-driven tech company, the central challenge is knowing what to build and why, and shipping at breakneck speed without giving up basic scientific discipline: clear hypotheses, controlled comparisons, and evaluations that someone else could rerun and verify. That’s meant treating scientific methodology less like a toolkit we can copy out of academia and more like a compass: knowing when to run a full evaluation and when a cheaper approximation is enough, when to dive deep into error analysis and when to stop and resurface, and how to keep results reproducible even as the underlying models shift under our feet.

The context problem

A researcher queries "BRCA1 mechanisms.". Vector search returns a mix of overviews and recent papers, all topically relevant. But if that researcher just read three advanced papers on DNA repair pathways, the context is mechanistic depth, not a primer. Change the query to "most recent evidence on BRCA1" and the user needs 2023-2024 papers, not a highly cited 2012 review. A vector embedding has no concept of publication date. Standard retrieval returns the same ranked list for both.

This pattern shows up across every domain. A journalist needs the most recent filings, not a company profile. An analyst needs quantitative depth, not a primer. An agent querying a news archive needs to know whether "latest" means today's coverage or the definitive investigation from last month. The topic is the same but the context is completely different, and the context determines which result is actually useful.

Why existing benchmarks can't capture this

This contextual collapse is why standard benchmarks like MS MARCO are insufficient for our domain. They assume relevance is a static, intrinsic property of a document. In high-stakes, specialized domains, relevance is a dynamic manifold. The same document can be exactly right for one user and completely wrong for another. A static test collection with binary labels has no way to express this.

Then there's the moving target problem. A retrieval pipeline that fails an eval in January can ace the same eval in March after a quiet model update, with no explanation of what changed. Offline evaluation assumes a stable system being measured against a stable ground truth. In practice, neither is stable. By the time a benchmark is published, the system it measured has already shifted.

And these test collections were designed for open web retrieval over massive indexes. Redpine isn't doing web search. We're retrieving from proprietary and licensed datasets across multiple domains. The distribution of queries that users and agents ask, the types of documents they need, and the criteria that determine whether a result is actually useful look nothing like a set of Bing searches from 2018 which MS MARCO is built from.

This forced us to build our own evaluation methodology at Redpine: a scoring methodology where each query-document pair is assessed not just for topical relevance but for context fit. We evaluate across distinct context classes, each with different criteria for what constitutes a good result, and track how each stage of the pipeline contributes. This lets us see exactly where value is added rather than just getting a single scalar that tells us we're "better" without explaining why.

How CAAR works

Context-Aware Adaptive Reranking (CAAR) is Redpine’s four-stage pipeline for making ranking functions conditional on query context.

First, the system classifies the query into a small set of context classes using a lightweight distilled model. This is conceptually similar to classic intent classification, but focused on how ranking criteria should change rather than on mapping to a product or facet.

That context class determines how downstream signals are weighted when reranking an existing candidate set: publication date for recency-critical queries, keyword specificity and domain terminology for depth queries, source reputation, impact and citation structure for authority queries. The same set of retrieved candidates can be reordered dramatically once you model what the user is trying to do.

Crucially, CAAR operates on the candidates you already have. It does not require additional retrieval calls, larger models, or a blow-up in latency. The full pipeline, including context classification and reranking, runs in under 50 milliseconds end-to-end, which keeps it compatible with interactive agents and tooling. And because it is retrieval-path agnostic, it can sit on top of vector search, sparse search, knowledge graph traversal, or hybrid stacks.

The data problem underneath

Retrieval methodology is only half the story. The other half is access to data that is actually worth retrieving.

In most specialized domains, the most valuable knowledge does not live on the open web. It sits in proprietary databases, licensed archives, and premium content from organizations that invest real resources in accuracy and quality; newsrooms producing original journalism, data providers curating structured datasets, publishers maintaining verified knowledge bases.

Web search barely scratches the surface of these sources, often only exposing short snippets of paywalled content instead of the full text. By aggregating and licensing multiple non‑web and paywalled data sources, we enable search inside collections that have effectively been unreachable for AI tools until now.

The AI ecosystem has often treated this content as raw material to be scraped, embedded, and summarized without compensation. That might work in the short term, but it erodes the economic foundations of the sources we depend on. When organizations cannot sustain high‑quality data production, every system that depends on that data degrades, no matter how sophisticated the models are.

Context‑aware retrieval makes high‑quality proprietary content more valuable because it can surface the right piece of that content under the right conditions, when depth, recency, or authority actually matter. In other words, better retrieval increases the marginal value of better data. Fair licensing and compensation give data owners a reason to keep investing in quality. One without the other does not hold.

Redpine is built on data partnerships, not web scraping. We unlock proprietary, premium content for AI companies and agents through a licensing model that ensures data owners and rights holders are compensated fairly.

What Redpine has built

Redpine is the infrastructure that unlocks proprietary data for AI, with retrieval that understands context and economics that keep the best data sources alive.

On the retrieval side, that means ranking that understands not just what a query is about but why it is being asked, and that can adapt criteria such as recency, depth, authority, and exploration accordingly without breaking latency budgets. On the data side, it means building the relationships and licensing structures that keep the highest-quality sources available and economically sustainable.

The systems that matter over the next few years will be the ones that:

• Treat relevance as a context-conditioned relationship rather than a static label
• Have reliable access to the best domain-specific data, not just whatever can be scraped, and
• Apply enough methodological rigor to know, with evidence, whether any of this is actually working in their setting.

That is the standard we are building Redpine upon.

‍

[Illustration: A simulation of the formation of dark matter structures from the early universe until today. Ralf Kaehler/SLAC National Accelerator Laboratory, American Museum of Natural History]

‍