When Do AI Outputs Tip?

Reversing “red or blue” to “blue or red” moved a model's probability of answering red by 65.3 percentage points on average. Its mean probability of an incorrect answer moved by only 1.1 points.

Across sixteen matched pairs, the shift toward blue helped when blue was correct and hurt when red was correct. The two effects largely cancelled in the average error. The probe was also too unreliable to study the loss of previously reliable behaviour, so I moved to a separately qualified retrieval task. Option-order sensitivity and forgetting need different measurements.

The broader project asks what a model carries into its next answer. Can a small description of its computation predict several future answers and the effect of a correction? Neil F. Johnson and Frank Y. Huo's Competition for attention predicts good-to-bad tipping in AI gave me a starting model: competing representations can change which output dominates as a conversation develops. ^[1]

What counts as tipping?

A model that was wrong from the start has not lost a reliable ability. Before looking for a transition, the retrieval task has to establish that the model can answer the source question under the specified wording and decoder. The earlier option-order probe failed that purpose. The later qualification fixes a narrower setting in which an incorrect conversational history can be compared with a correct one.

There are also different events hidden inside the word tipping. The most likely answer can switch when two scores cross, even while both probabilities change smoothly. A sampled conversation can make a sudden first error without any abrupt change in its underlying probabilities. Persistent wrong answers, recovery after an error, and a change in factual topic are further outcomes. A first-error clock addresses only the first failure along the declared continuation.

These experiments change inference-time state with the model weights held fixed. They concern what generated history and cached computation do to later answers. They do not measure catastrophic forgetting from training on a new task, although the question of preserving some readouts while changing others connects the two problems.

Which level of analysis answers the question?

A semantic label is often the right level for evaluating an answer, but it can hide the token-level differences that determine the next one. Conversely, a detailed attention map can show where information was read without establishing that the final answer changed. Following the computation from an internal edit through the vocabulary distribution to accepted answer sequences connects those levels.

The full cache contains much more information than a single risk score, but retaining everything gives little explanation of what matters. The aim is a smaller state whose update and predictions remain valid for a stated family of questions and interventions. Its adequacy depends on the target: predicting the next error, predicting its timing, and choosing a correction can require different information.

Swapping one head's message

Qwen2.5-0.5B-Instruct read a source fact and answered a factual question. I kept an incorrect earlier answer in the visible conversation, then replaced one attention head's internal message with the message from a matched conversation containing the correct earlier answer.

On sixteen held-out prompts, mean error probability fell from 3.65% to 1.33%. The selected head was layer 16, head 7, using zero-based indexing; selection used separate discovery prompts. These probabilities came from a decoder restricted to two answer tokens. They are conditional model probabilities, not error frequencies from sampled conversations. ^[2]

The message has two components worth separating: the values carrying information and the attention weights deciding how to combine them. Replacing only the value content, while keeping the recipient's attention weights, reduced mean error to 1.26%. Replacing only the attention weights gave 3.70%. Content transfer reproduced the average rescue in this experiment; routing transfer did not.

The matched donor makes this a causal test of a specific internal replacement. Separating content from routing narrows the explanation further: a change in the weighted message can come from different information being available, different weights on that information, or both. The comparison tests those replacements while keeping the other component fixed.

The location of the edit also matters. In this cached decoder, temporarily patching the final layer's attention output can change the current logits without changing stored keys or values. If the same next token is forced in both runs, their later computations receive the same cache. An edit to stored values can persist because later queries can read those values again.

The rescue does not establish that this head is the sole natural cause of the error. A donor replacement may introduce a message the recipient would not normally produce, and other heads or downstream operations can compensate for it. The held-out prompts reduce the risk of reporting only the prompts used to choose the head; they do not establish transfer to another model, an unrestricted conversation, or a different intervention. The prompt-level plot also matters because an average can conceal opposite effects across source colours and wording.

An attention mass for the response clock

My work on continual learning and physical modelling keeps running into a related problem: a variable can describe the present accurately while omitting information needed to update it. The fixed-prototype binary attention model gives a small example.

Let $F$ be the current score margin for the incorrect answer, so its probability is $\sigma(F/\tau)$, where $\tau$ is the decoder temperature. The margin fixes the next-answer probability. The effective attention mass $\nu$ controls how strongly another correct output changes that margin. After $j$ additional correct outputs,

$$F_j = \frac{\nu F + j\beta}{\nu + j}.$$

Here $\nu$ measures existing attention weight in units of one additional correct output. The fixed output prototypes determine the limiting margin $\beta$. With $\nu=4$, the next correct output receives one fifth of the updated attention mass. With $\nu=40$, it receives one forty-first.

Two histories can therefore start with the same error probability and evolve differently. At future answer $j+1$, the hazard is conditional on the preceding answers being correct. Multiplying the survival probabilities gives the chance of a first error within $H$ answers:

$$\Pr(T\leq H)=1-\prod_{j=0}^{H-1}\left[1-\sigma\left(F_j/\tau\right)\right].$$

This is an exact construction for the stated simplified head, with fixed prototypes and no positions, residual path, MLP, or layer normalisation. It is not a fitted law for Qwen. The neural experiments still have to establish which parts survive the additional computation.

The extra mass variable explains why a current risk score can be insufficient even when it is measured exactly. Repeating every item in a history can preserve its weighted average while increasing the mass already present. One new output then moves that average less. In the construction, the current margin describes where the system is; the mass describes its resistance to the next update.

For a negative initial margin and a positive limiting margin, a greedy answer switches when accumulated updates carry the margin through zero. Increasing the initial mass delays that crossing. At nonzero decoder temperature there is no single guaranteed failure time: an error can occur before the crossing or a correct answer after it. The cumulative curve integrates those conditional risks. Its survival product does not assume independent errors.

The restriction to the first error makes the state update unusually simple. Before that event, every generated answer belongs to the correct prototype. Once an incorrect answer occurs, future queries and stored contributions can change. Predicting recovery or total errors then requires additional response information. Two histories can agree on every first-error probability and still differ after that first error.

The clock also suggests a test that can fail. With a fixed temperature, the first three conditional hazards can identify a nonconstant candidate clock; the remaining horizons then test its predictions. Fitting each horizon separately would not test the update law. Near-constant hazards make parameter estimates unstable, and agreement would still leave open whether several neural mechanisms can produce the same curve.

The time of a correction

Timing matters in the measured model too. In one recorded history, re-encoding an earlier correct “Red” as “red” before a two-answer continuation improved net success by 3.31–3.70 percentage points. Applying the corresponding normalisation only after a later accepted “Red” answer had a small negative effect on whole-continuation success.

That comparison was calculated post hoc from measured conditional probabilities. It shows why a correction needs a position in the sequence as well as a description of what it changes.

“red”, “Red”, and a longer sentence can express the same answer but leave different token histories. Semantic correctness sums the probabilities of accepted complete sequences. A ranking of one representative vector need not rank that sum.

For the later experiments I defined twelve accepted answer forms, included their end-of-answer tokens, and retained the original vocabulary normalisation. Unmeasured branches kept explicit probability bounds. Those ranges describe the contribution missing branches could make; they are not sampling confidence intervals.

A two-token decoder deliberately removes much of that complexity. Renormalising over red and blue measures preference conditional on producing one of those tokens. It does not count the possibility of another word, a longer answer or an unfinished response. That makes it useful for a controlled intervention comparison, but its error percentages cannot be read as unrestricted chat failure rates.

Likewise, a correction that preserves meaning can change later behaviour through its exact spelling and tokenisation. Measuring only semantic correctness at the time of the edit would miss that difference. Measuring a single next-token probability could miss a change elsewhere in the completed answer. The answer contract and evaluation horizon therefore belong in the experiment specification.

Accurate prediction, a difficult baseline

A 64-number description of a structured value-cache edit, together with reference computation retained from the decoder, predicted 296 future conditional vocabulary distributions. The largest total-variation discrepancy was about $1.4\times10^{-9}$. That prediction still required the original nonlinear downstream operations: the 64 numbers describe only the edit. The CPU decoder read trained BF16 weights and computed in float64.

A state-dependent correction policy improved two-answer success by roughly 0.052–0.070 percentage points over retaining the edited cache. A simpler rule, given the current complete-answer probability, chose the same correction in every tested state. Accurate prediction had not yet demonstrated a decision advantage from remembering the intervention.

Prediction accuracy and decision value are separate tests. If two states have different future risks but the same best available action, distinguishing them does not improve that choice. Conversely, a small risk difference can matter when a repair has a cost close to its expected benefit. The action set, fee, horizon and failure loss determine which state differences are useful. This motivation is related to value equivalence in model-based reinforcement learning, where a model is judged by the planning quantities it preserves. ^[5]

Holding the current answers fixed

The next test changes cached state while preserving declared current readouts. A content vector is distributed across 32 cached positions. Three current queries across seven reader heads impose 21 scalar constraints on that position profile. The remaining freedom can suppress current head-message changes while retaining a response at later public queries.

This uses the same geometry that makes protected subspaces useful in continual learning: a change can be invisible to one set of readouts and visible to another. The edited cache values were rounded to float32 before checking the actual output probabilities.

Four contexts combined red or blue source facts with positive or explicitly negated wording. Two opposite edits per context produced eight internal states. All eight passed the frozen matching checks:

Checks of the accepted spellings separately also found very little redistribution. Internally different states now agree on the measured current behaviour to high precision. The future semantic-error comparison is unfinished. A later head's nonzero response alone does not establish a later change in semantic correctness.

The observation used by a controller therefore needs to be specified. A head message, a vocabulary distribution, and the meaning of a complete answer preserve different information. Nick Cammarata's Goodfire essay, Neural Networks Are Already Clean, argues for examining native mechanisms with their inputs and downstream readers. Its broader claims are a working hypothesis; following the actual downstream computation is useful for these experiments. ^[4]

The next comparison gives one controller the edit description and others the matched initial readouts. It also includes a controller allowed to measure risk later and react then. If that reactive controller catches up, retaining internal state would need to justify its value through observation cost, latency, or an intervention window that closes before the next measurement. I have not established that advantage yet.

The position-profile construction provides a way to ask that question without merely comparing two visibly different prompts. Current readers impose linear constraints on how a value edit is distributed across cached positions. A direction left unobserved by those readers may become visible to a different query later. Passing the linear constraints is only the construction step; checking the actual vocabulary and complete-answer probabilities after rounding is what tests whether the intended matching survived the rest of the network.

Current matching has a precise scope. Agreement on three queries does not imply agreement on every question a controller could ask, and a later internal response does not by itself establish a semantic failure. A useful follow-up has to fix the observation interface before evaluation, test later complete answers, and compare correction outcomes on fresh histories. Otherwise the observer can keep acquiring measurements until the claimed hidden difference is no longer hidden.

What would make the extra state useful?

The strongest result would be a pair of states that a declared observer cannot distinguish, yet for which the available repairs have different best choices. That would show a limitation of the observer for this decision, rather than merely another difference inside the network. If one repair is best in both states, the observer may remain adequate despite predicting their risks imperfectly.

A state-aware controller should therefore be compared with keeping the cache, always applying the repair, a rule using the current complete-answer risk, and a controller allowed to measure again later. They need the same repair options and the same accounting for measurement and intervention costs. Any gain also has to survive new prompts, new histories and changes in the allowed edits.

The practical motivation is to correct a conversation before an error becomes expensive. These experiments establish a local causal rescue and a mathematical example of missing temporal information. The matched-readout experiment is an attempt to find the corresponding decision problem in a trained model. Whether its extra state provides useful advance warning remains the question being tested.