LLM Probability Concentration: How Alignment Shrinks the Generative Horizon

A unified view of LLM probability concentration

Why are aligned language models insensitive to decoding choices, why do long reasoning traces produce stable answers, and why does late resampling often fail? We connect these behaviors through probability concentration in the model's output distribution.

We introduce Branching Factor (BF), a token-invariant measure of the effective number of plausible next steps during generation. The camera-ready analysis yields three complementary findings:

Together, these results separate a task-independent autoregressive self-narrowing tendency from the additional concentration induced by alignment.

BF is not a monotone token-wise law. It is an aggregate tendency that can be locally reversed by injecting context that is unexpected under the model's current trajectory.

This unified view explains why low-BF aligned models barely respond to decoding hyperparameters, why lower BF bounds gains from additional majority votes, and why late low-BF forks hurt accuracy after a model has already committed.

Longer CoT alone does not explain lower BF

We compare reasoning-oriented DeepSeek-R1-Distill-Llama models with direct-answer Llama-3-Instruct models at identical truncated output lengths on MMLU. The reasoning models have lower BF at every matched position, including the earliest ones, so their concentration is not merely an artifact of longer generations reaching later low-BF regions.

Matched-length BF comparison. Reasoning-oriented models remain more concentrated than direct-answer aligned models at every shared truncation length.

Why BF decreases—and how it can rise

A falling BF curve alone does not identify its cause. We therefore replace the model's own prefix with equally long, independently sampled random tokens while keeping the model fixed. BF surges at the substitution point and then falls again as autoregressive generation resumes. The same qualitative pattern appears across base/aligned pairs: alignment sets the absolute band and re-collapse speed, while self-conditioning drives the broad downward tendency.

External random-prefix intervention. Dashed lines mark the substitution points. Unexpected external content reopens the branching space; continued self-conditioning narrows it again.

A second route to higher BF

Unexpected environment feedback

In matched one-turn agentic scenarios, only the Environment Feedback: field changes. Adversarial feedback invalidates part of the current plan; random noise occupies the same prompt slot. Both raise next-step BF relative to progress-as-expected controls, with the effect compressed in more heavily aligned models.

Takeaway. Unexpected context can reopen the consideration set, but continued autoregression tends to narrow it again.

Case Study: Is Decoding Method Still Crucial for Modern LLMs?

Many prevalent decoding methods were introduced before LLMs scaled to billions of parameters and underwent multiple training stages. Additionally, model developers adopt different decoding strategies when reporting LLM capabilities , raising questions about the significance of decoding choices for modern LLMs. To explore this, we benchmark various decoding methods on standard LLM reasoning tasks, extending prior work to the latest models including DeepSeek-distilled models , which would generate long CoT before the final answer. Specifically, we evaluate model performance on MMLU-STEM under CoT prompting across different temperatures ($T=0.6 / 1.0$) in temperature sampling and truncation thresholds ($p=0.9 / 1.0$) in nucleus sampling .

The results in Table 1 reveal that for aligned models, decoding configurations have a limited impact -- typically around 10% (up to 20%) relative performance changes. Among Llama-3.1-8B models, DeepSeek-distilled Llama-8B (based on Llama-3.1-8B), which is trained to generate long CoT, exhibits the smallest relative performance changes. In contrast, base models exhibit greater sensitivity, with performance varying by up to 31%. Additionally, lowering the temperature ($T$) generally improves performance across all models more than adjusting truncation threshold ($p$), though excessive reduction (e.g., greedy decoding when $T=0$) may lead to repetition issues .

Based on these observations and findings in existing literature, we propose the following hypotheses:

Researchers often assess probability concentration using token-level metrics such as entropy or log-likelihood. However, these offer only a narrow lens on model behavior: they capture local properties but miss the global structure of the output space--how probability mass is distributed across plausible sequences. This motivates our proposal of the BF as a structural measure of generative breadth.

Measuring Branching Factor

The generative process of language models can be viewed as moving down a branching tree, with each token choice selecting a path forward. While the full tree spans $O(|V|^N)$ sequences for vocabulary size $|V|$ and sequence length $N$, LLMs concentrate probability mass on a far smaller subset. To capture how many options the model seriously considers at each step, we introduce the Branching Factor (BF). Given $|\mathcal{T}|$ high-probability leaf sequences, we approximate the tree as a balanced $B$-ary tree, where $B = |\mathcal{T}|^{1/N}$. In this section, we describe how to compute $|\mathcal{T}|$ and $B$ in practice.

Intuitive Perspective: Exponentiated Entropy (Perplexity) as Branches

We propose to use the exponentiated entropy (perplexity) to quantify $|\mathcal{T}|$: $|\mathcal{T}| \stackrel{\text{def}}{=} \exp\left({H}(Y_{1:N} | x; \theta)\right)$. This reflects the effective number of equally probable outcomes with the same total uncertainty . Analogously, it is like sampling from a fair $|\mathcal{T}|$-sided die, where entropy equals $-\sum \frac{1}{|\mathcal{T}|}\log {\frac{1}{|\mathcal{T}|}} = {H}(Y_{1:N} | x; \theta)$. Thus, $B(x;\theta) = \exp\left({\bar{H}(Y_{1:N} | x; \theta)}\right)$ where ${\bar{H}(Y_{1:N} | x; \theta)}=\frac{1}{N} \tilde{H}\left(Y_{1:N} | x; \theta \right)$ is the averaged entropy per output token up to position $N$. A larger $B(x; \theta)$ indicates a greater potential for diverse outputs.

For short outputs, where it's tractable to sample sufficiently many sequences to closely estimate the conditional entropy at each position, we can estimate the BF by computing the conditional entropy at each position and then aggregating as:

where $\tilde{H}(Y_t | [x, y_{1:t-1}^{(i)}]; \theta)$ is the entropy of the distribution at position $t$ for sample $i$.

Practical BF Estimator via Asymptotic Equipartition Property

While the above approach works well for short outputs, it becomes challenging for longer sequences, as we can only sample a tiny fraction of the exponentially large output space. In such cases, we show that when LLMs generate sufficiently long outputs, the average log-probability of each output sequence will be roughly the same, and can approximate average output entropy well, following the Asymptotic Equipartition Property (AEP) . The original AEP proof requires additional assumptions about the generation process, such as that it needs to be stationary and ergodic, often violated by LLMs. But as noted by , these assumptions are unnecessary if we do not require ${\bar{H}\left(Y_{1:N} | x; \theta \right)}$ to converge to a constant:

This theorem is equivalent to the statement: for sufficiently large $N$, the probability of any length-$N$ high-probability output $y_{1:N}$ under $\tilde{P}$ can be approximated as $\exp\left(-N\bar{H}(Y_{1:N} | x; \theta)\right)$, rendering log-probability asymptotically ineffective for distinguishing among them.

As an empirical demonstration, we plot the standard deviation of the average negative log-likelihood of Llama-3-8B-Instruct over multiple datasets in Figure 2, where we can see that with the increased output length, the difference between length-averaged entropy and negative log-likelihood (NLL) is reduced, and the standard deviation of average NLL also quickly reduces within the first 50 output tokens.

Therefore, for long sequences, we can estimate BF using the NLL of sampled sequences as:

This approach allows us to compute BF in a sample-efficient way. For task-wise BF, we simply compute it via averaging all instance-wise BF: $B(X; \theta) = \sum_{x} p(x) B(x; \theta)$.

Benchmarking and Attributing Branch Factors

In this part, we will introduce our BF computation experiment settings, including models, tasks, and the impact factors influencing BF.

Models and Sampling

We run experiments on models from Llama-2 and Llama-3 families as they are widely-used open-weight model families. For each model family, we include both base and aligned models to investigate how alignment tuning affects BF. We set $p=0.9$ and $T=1.0$ to sample outputs to conform with the setting for most datasets.

We set $M=50$ sequences to estimate BF, which yields a reliable estimation across datasets in prior studies. For aligned models, we apply the official chat templates to prompts. In addition, we carefully control the lengths of all inputs plus outputs to be within the context window of the models.

Tasks

We consider a variety of tasks covering common application scenarios of LLM generation, including reasoning and open-ended generation: MMLU (Reasoning), Cognac (Controlled Generation), BBCLatestNews (News Generation), and Creative Story Generation (Creative Generation). To test subjective randomness bias , we also prepare a synthetic task Random Strings where the prompt is generated via randomly sampled characters.

Impact Factors (IFs)

We consider modulating these factors that may impact BF computations: Prompt Complexity ($C$), Alignment Tuning $(AT \in \{\text{Instruct},\text{Base}\})$, Model Size $(S \in \{8\text{B}/13\text{B},70\text{B}\})$, and Model Generation $(G \in \{2,3\})$. $C$ controls the informativeness of the input prompt $x$ (e.g., the number of banned words in Cognac, the number of in-context samples in MMLU). Intuitively, providing more information in $x$ should make the model more confident in its outputs, resulting in a lower BF. $AT, S, G$ represent model-wise variations to explore how different configurations of $\theta$ affect $B(X; \theta)$.

BF Dynamic in Generation Process

Both BF and the output length $N$ are functions of the output $Y$, and BF computation relies on $N$. To avoid confounding effects, we first analyze how BF varies with $N$ before intervening IFs. We demonstrate BF trajectories over different output positions by running Llama-3-70B and Llama-3-70B-Instruct on three representative tasks. Specifically, we compute BF over every five output tokens, conditioning on the prompt and all previously generated output tokens.

Across tasks, alignment reduces BF by roughly 2–5× overall, with gaps approaching an order of magnitude at some early output positions (for example, approximately 12 versus 1.2). This leaves fewer candidate tokens for decoding to affect. BF also tends to decline as generation proceeds, although the controlled interventions above show that this is an aggregate self-conditioning tendency rather than a monotone token-wise law.

Pareto Analysis of BF

We perform a Pareto analysis to identify the relative influence of all IFs of BF. For each factor $D_i$, we define the unnormalized Impact $\tilde{I}(D_i)$ as the average absolute pairwise difference in BF when varying $D_i$ while holding other dimensions constant:

Then we normalize it as ${I}(D_i)=\frac{\tilde{I}(D_i)}{\sum \tilde{I}(D_i)}$.

The results indicate that alignment tuning is the most influential factor affecting BF, surpassing model size, model generation, and prompt complexity by a large margin. For tasks with richer inputs--such as MMLU (with more in-context examples) and BBCLatestNews (with more headlines)--prompt complexity $C$ and model size $M$ emerge as the next most impactful factors. In contrast, for open-ended tasks like Cognac and Story Generation, model generation $G$--particularly improvements from Llama-2 to Llama-3--plays a more dominant role. This shift likely reflects gains from the use of larger, more diverse datasets in training .

Curious Case of Prompt Complexity

Intuitively, greater prompt specificity (larger $C$) reduces BF by narrowing the model's output space through more informative context. However, our experimental results reveal task-varied effects. For the Cognac task, greater prompt complexity can increase BF--potentially due to the cognitive burden of processing negation or complex linguistic structures. In contrast, for tasks like News Generation, higher $C$ generally leads to lower BF, consistent with the expected narrowing of output diversity.

Application: Variance Reduction and Risks of Mid-Generation Forking

Building on our findings that BF declines over the generation process and is lower in aligned models, we derive a practical implication: aligned CoT models, by starting with low BF and delaying decisive tokens, shrink the output space more aggressively and produce fewer high-probability variants. To test this, we evaluate output variability on MMLU-STEM using 200 samples per model, measuring the standard deviation of Majority@K accuracy for $K = 1, 3, 8, 16$ under temperature $T=0.6$ and truncation threshold $p=0.9$.

As shown in Table 2, among models with similar capacity, those with lower BF--especially the aligned CoT model--exhibit markedly lower variance. This confirms that BF is a reliable predictor of sampling stability.

How does Alignment Tuning Impact BF?

Why does alignment tuning exert such a pronounced effect on BF? Building on the superficial alignment hypothesis ("Alignment tuning might simply teach base LLMs to select a subdistribution of data formats for interacting with users.") and recent work on tuning-free alignment , we hypothesize that base models already encode low-entropy conditional distributions. In this view, alignment tuning doesn't reshape generation from scratch, but instead nudge the model toward stylistic tokens (e.g., "Sure"), thereby narrowing the conditional output distribution.

To test this hypothesis, we reproduce the nudging experiments , over Just-Eval-Instruct and MMLU datasets. We employ Llama-3-70B for drafting most outputs. However, when the base model's Top-1 probability is low, we apply nudging by switching to Llama-3-8B-Instruct to generate a single word. BF was computed as in prior experiments.

The results indicate that after nudging occurs early in the generation process -- indicating the prefix generated by the nudging model is of low probability. These observations collectively support our hypothesis. Considering that nudging not only reduces BF but also improves aligned model performance on these tasks , our results highlight the dual effect of alignment training: reducing BF while preserving or even enhancing task performance.