How Arcana indexes the live web in real time

The problem with static indexes

Traditional search engines crawl the web on a schedule. Pages are fetched, indexed, and stored. By the time you run a query, you are searching a snapshot of the web from hours, days, or weeks ago. For evergreen content, this is acceptable. For anything time-sensitive, it is a fundamental limitation.

Arcana was built on a different premise: every query should search the web as it exists right now.

How real-time indexing works

When you submit a query to Arcana, a live search process is triggered against the current web. We do not retrieve pre-indexed results. Instead, Arcana identifies the most relevant domains and sources for your query, fetches their current state, extracts the meaningful content, and synthesizes a response. This happens in under 1.5 seconds on average.

Source selection and trust scoring

Not all sources are equal. Arcana maintains a continuously updated trust graph that scores domains across editorial quality, factual accuracy history, update frequency, and citation patterns from other trusted sources. When composing an answer, Arcana weights sources by their trust score for the specific domain of the query.

Why this matters for accuracy

Stale indexes introduce a subtle but significant accuracy problem: the answer that was correct six months ago may no longer be correct today. Regulations change. APIs deprecate. Research findings get updated or retracted. By searching the live web on every query, Arcana eliminates this class of error entirely.

What comes next

We will share more on the technical architecture of our source fetching and synthesis pipeline in a future post, including how we handle conflicting sources and low-quality content at scale.