When confidence becomes a teaching tool, the risk is not just wrong answers

Confidence can turn error into memory

The new arXiv paper changes the problem framing in a useful way. Its central finding is not simply that a chatbot can state falsehoods with confidence. It is that a confident, consistent answer can become something users carry away as knowledge. In a within-subjects experiment with 32 participants, the authors had learners chat with GPT-5-mini assistants whose briefings were quietly seeded with plausible falsehoods. Across 104 quizzes and 1,040 answers, scores fell from 7.4 to 2.9 out of 10, and the planted option was chosen on 60% of falsified items.

The more consequential result is in the mechanism. The paper reports that whenever the assistant stated a falsehood, it was adopted about 65% of the time at every falsification level. Absorbed falsehoods were then held with high confidence, and the modelled confidence advantage of correct over wrong answers disappeared. That suggests the risk is not only mistaken output, but mistaken output that helps users feel settled. Once that happens, confidence stops acting as a warning signal and starts acting as a delivery system for the error.

That mechanism is also why the study’s authors argue the issue is not just model sycophancy. They say the assistant was almost never sycophantic and never conceded under pushback, yet the conversation still produced human deference. The model did not need to flatter the user for the user to absorb the wrong answer. A fluent, assertive explanation was enough to create a false sense of understanding, and that sense of understanding can suppress the impulse to check.

A plausible alternative explanation is that the experiment makes deferral look cleaner than it would in ordinary use. The sample is small, the task is narrow, and participants were not allowed to consult outside sources. That means the study does not show how people behave in a messy, open information environment. It does, however, show something more specific and more troubling for design: a single confident source can make verification feel unnecessary even when the source is wrong.

That matters most in settings where the same system both teaches and validates. Education is the clearest case, but the implication reaches any workflow that treats an assistant as both answer engine and confidence cue. The practical response is not just to demand better accuracy in the abstract. It is to make reliability legible claim by claim, so users can separate what was said from how sure it sounded.

The next signal to watch is whether independent verification breaks this loop. If users become less willing to adopt confident falsehoods when they can check against another source, the thesis strengthens: the real hazard is self-authenticating error. If they do not, the problem may be broader than this paper suggests, and confidence would be only one part of a deeper deference pattern.

Source: https://arxiv.org/abs/2610.31579