Confidence scoring: how Arcana knows what it knows

The problem with confident-sounding wrong answers

AI systems have a well-documented failure mode: they produce confident-sounding answers even when their underlying evidence is weak, conflicting, or absent. This is particularly dangerous in research contexts, where a wrong answer stated with authority can propagate into decisions, documents, and downstream analysis.

Arcana approaches this differently. Every answer includes a confidence indicator that reflects the quality, recency, and consensus of the underlying sources.

How confidence is calculated

Arcana evaluates four dimensions for each answer it generates. Source quality scores the trustworthiness of the domains contributing to the answer, based on our trust graph. Source consensus measures how much agreement exists across independent sources. Recency reflects how current the information is relative to the query's time sensitivity. Coverage measures whether the sources collectively address the full scope of the question or only part of it.

These dimensions are combined into a single confidence signal displayed alongside the answer.

What low confidence means in practice

A low confidence indicator does not mean the answer is wrong. It means the evidence base is thinner than usual. This can happen when a topic is very recent, when sources disagree significantly, or when the query touches areas where high-quality sources are scarce.

In these cases, Arcana surfaces the disagreement explicitly rather than picking a side. You see what the sources say, where they diverge, and why certainty is limited.

Why this matters for research workflows

Knowing when not to trust an answer is as valuable as getting a good answer. Researchers who know which parts of their analysis rest on weak evidence can focus their verification effort where it actually matters, rather than treating all AI output with uniform skepticism or uniform trust.