The case against training on user data

A common practice worth questioning

Most AI products improve their models by training on user interactions. When you use a product, your inputs and the model outputs become training data for the next version. This is presented as a feature: the product gets better the more people use it.

We think this framing obscures a significant cost to users.

What training on user data actually means

When a company trains on user queries, your searches become part of the dataset that shapes model behavior for all future users. In aggregate, this reveals a great deal about what people are curious about, worried about, researching, and deciding. For search specifically, queries are among the most revealing data a person generates.

The value exchange is real but asymmetric. The user provides sensitive behavioral data. The company receives a better model. The user receives a marginally improved product experience, with no control over how their data contributed to it.

The consent problem

Training data consent is typically buried in terms of service. Users who read those terms carefully and understand the implications are a small minority. Consent obtained through terms most users do not read is not meaningfully informed consent.

The leakage risk

Models trained on user data can inadvertently memorize and reproduce specifics from that data. This is a known and studied phenomenon. For search queries that include sensitive personal, professional, or financial information, the risk that this information could be surfaced through model outputs is not theoretical.

Our position

Arcana does not use user queries to train models. We improve our systems through evaluation on synthetic and publicly available benchmarks, through red-teaming, and through structured feedback mechanisms that users explicitly opt into. The tradeoff is that our models improve more slowly than they would with user data. We think that is the right tradeoff.