Distinguishing Forms from Estimators in Interp

Why did you steer by difference-of-means?

Is there a direction in my model's activations that mediates safety? Or honesty, or diligence (Zou et al., 2023). The template of this question, is there an X in my model that mediates Y is a common start to lines of interpretability thinking. The higher-level goal of these questions is often to discover useful hooks/interventions/monitors one can apply to models to shape or audit model behavior. But the methods we use—steering vectors, SAEs, neuron analysis, etc.—basically are just opinionated versions of finetuning methods, like LoRA or full finetuning: you pick how you're allowed to change the model (e.g., by steering, or LoRA) you choose the particular parameters (estimation) and then you evaluate the efficacy and specificity of the resulting intervention.

This note is on how to do a good job of answering interpretability questions rigorously by remembering to decompose the question of the existence of \(X\) into what kind of intervention we're searching for (steering vectors, neuron selection, high dimensional sparse latents, etc.) and estimation: the way we choose the particular member of that family (difference-of-means, autoencoding loss, DPO, cross-entropy,…) After making a brief argument about why we might consider estimating steering vectors by gradient descent instead of difference-of-means, I'll run through some experiments comparing estimation methods. Overall, I find that gradient-based estimators tend to outperform difference-of-means for causing model behaviors, especially when you care about output quality, not just steering efficacy.

These are not new ideas: already, Zou et al. (2023) and Cao et al. (2024) optimize steering vectors by gradient descent. But I want the steering experiments here to be just a case study for a broader point: we should be separating the form of our interpretability tool from its estimator always, whether in sparse activation decomposition, activation explanation, neuron analysis, anything.

Overview.

You can think of interpretability/intervention/discovery hooks as having two components in a classic machine learning sense: the form and the estimator.¹ Let's take steering vectors as a concrete example. The form is something like add a vector \(v\in\mathbb{R}^d\) at layer \(\ell\) at every token index in the sequence and run the rest of the forward pass of the network, that is, it's something like

We're sort of skipping some details here, but roughly, the form is well-described by the layer you're intervening at, the shape of the vector, and the process of how to perform the intervention.

The estimator, on the other hand, is how we choose \(v\) (and to a lesser extent, \(\ell\)!) For example, we might have some data representing sycophantic responses to a question, and some data representing non-sycophantic responses. We might freeze \(\ell\) somewhere two thirds of the way through the layers of the network (maybe \(19\) out of \(32\)) and then estimate the steering vector via a difference-of-means:

where the two \(\mu\) are constructed by taking the average activation under the network at that layer for the corresponding dataset, \(\mathbb{E}[h^{(\ell)}]\).

We then evaluate how well our intervention actually causes sycophantic behavior, how it affects the rest of the model's distribution, etc. Evaluation is certainly very hard, as is the process of selecting the data on which to form our estimator, etc. But in this note we focus on one question that comes up frequently to me as I advise safety and interpretability research: why did you choose difference-of-means as your estimator of \(v\)?

I hear many answers to this question, from The difference of means represents the concept direction under the model, which is an interesting idea but seems like a claim to be tested rather than a premise, to It's simple to compute and works, which I wholeheartedly agree with. To me, it's fascinating that difference-of-means often works, and often works well (Rimsky et al., 2024). But there are other estimators out there; for this example, we could estimate \(v\) by gradient descent using a preference loss like Direct Preference Optimization (DPO) (Rafailov et al., 2023); being a bit free with notation we might write this as an expectation over the dataset \(D\) of syco/non-syco responses of the DPO loss of the model (under intervention by \(v\)):

so, we pick Adam or SGD or something and some hyperparameters for our optimization process (early stopping, maybe the \(\beta\) in DPO, etc.)

On the one hand, this is clearly more complicated than DoM. On the other hand, we get to specify what we want to find: a vector \(v\) that makes the model do sycophantic (as opposed to non-sycophantic) stuff. If we're particularly worried about a specific side-effect, say, whether our intervention actually also makes coding quality worse, we can simply add a proxy for that to our optimization:

where our proxy here is something like a token-level KL-divergence between the original model's distribution and the distribution post-intervention, as taken over a distribution of coding text.

While optimization is more complicated, it also lets us be more precise about what we're looking for, and the whole of deep learning has built us tools that make the complexity easy to navigate: our modern variants of gradient descent just work, and tools like PyTorch make them very simple to work with.

In the next few sections, I'll show some experiments comparing DoM to various gradient-based estimators for steering vectors. I'll also discuss what it means that, well, a bunch of these estimators work (some better than others!) and they come up with sometimes rather different steering vectors.

Simple experiments with steering vectors

We'll go through a few simple experiments to test the effectiveness of steering vector estimations across steering efficacy—whether they can perform the steering necessary—and quality—the mean of GPT-6-Luna's coherence and relevance scores, each on a 1–10 scale.

Datasets and metrics

We're going to evaluate our steering estimators on four datasets/steering objectives: sentiment, translated output (French), sycophancy, and a 3-way if/else conditional code generation dataset in which models are intended to use Google-style docstrings if a question relates to game dev, a Sphinx-style docstring if the question relates to web dev, and NumPy-style docstrings if a question relates to AI. Outputs for all tasks' training sets are generated by prompting a model for the chosen and rejected outputs. For generating these prompted responses, we use GPT-6-Luna for our first experiments. We then later use the target models themselves (prompted) to generate the training datasets, to see if this makes a difference.

Sentiment. A simple sentiment task—input is a question from Alpaca (Taori et al., 2023), chosen output is a positive-sentiment response, rejected output is negative. Steering efficacy is measured as LLM judge positivity of the model's generated output, 1-10 rescaled to 0-1 as \((\text{positivity}-1)/9\).

French. Just generate French output! Again, inputs are Alpaca questions (Taori et al., 2023), chosen output French, rejected output English. The steering efficacy is fastText's estimated probability of the output being French (Joulin et al., 2017). (Lol @ some LM for choosing fastText instead of using LLM-as-a-judge, saving me a few tokens.) The judge is also told that the output is expected to be in French despite the input being in English.

Sycophancy. Models are tasked with generating sycophantic text (Sharma et al., 2023). Half of the texts (at training and eval time) are trivia where the user states an incorrect answer and half are requests for feedback wherein the user claims to write the text. Chosen responses either agree with the user's incorrect answer or praise the text written by the user. Rejected responses, correct the user or give a real review. For steering efficacy, we measure both the tendency to agree with the user's incorrect answer as well as the level of praising the user's text.

Conditional Code Style. These inputs are coding requests: 1/3 web, 1/3 gaming, 1/3 machine learning. This task is a bit odd—it's a 3-way if/then statement—if the query is a web query, generate code with a Sphinx-style docstring. If machine learning, then NumPy-style, and if game dev, then Google-style. This is interesting because unlike the other steering goals, the expected change in behavior of the model is dependent on the input.

def sort_inventory(inventory):
    """Sort inventory items by value while preserving their quantities.

    Args:
        inventory (dict): Mapping of item names to quantities.

    Returns:
        list: (item, quantity) pairs sorted by item value.
    """
    [...truncated]

def sort_inventory(inventory):
    """Sort inventory items by value while preserving their quantities.

    Parameters
    ----------
    inventory : dict
        Mapping of item names to quantities.

    Returns
    -------
    list
        (item, quantity) pairs sorted by item value.
    """
    [...truncated]

def count_token_frequencies(text):
    """Count whitespace-delimited token frequencies in a text corpus.

    Parameters
    ----------
    text : str
        The corpus to tokenize.

    Returns
    -------
    dict
        A dictionary mapping each token to its frequency.
    """
    [...truncated]

def count_token_frequencies(text):
    """Count whitespace-delimited token frequencies in a text corpus.

    :param text: The corpus to tokenize.
    :type text: str
    :returns: A dictionary mapping each token to its frequency.
    :rtype: dict
    """
    [...truncated]

def build_query(params):
    """Build a URL query string from parameter names and values.

    :param params: Mapping of parameter names to values.
    :type params: Mapping
    :returns: The URL-encoded query string.
    :rtype: str
    """
    [...truncated]

def build_query(params):
    """Build a URL query string from parameter names and values.

    Args:
        params: Mapping of parameter names to values.

    Returns:
        The URL-encoded query string.
    """
    [...truncated]

Estimators.

Let our dataset be composed as tuples of queries, chosen responses, and rejected responses, as \(D=\{(x^{(i)}, y^{(i)}_c, y^{(i)}_r)\}_{i=1}^{M}\).

DoM (mean). This steering vector estimator (Rimsky et al., 2024) averages together the activations for all tokens in the chosen response (conditioned on the query)—and then averages across all chosen responses in the dataset, and subtracts the corresponding average for the rejected response. Let \(h_{1},\dots,h_{T}\) be the activations for tokens \(1\) to \(T\) of a \(y\) of length \(T\) when processed by the model conditioned on \(x\). Then we have:

DoM (last token). One variant of the DoM method is to represent each chosen (and rejected) response as just the last activation instead of the mean of all activations. The estimator is the same except for the definition of \(\phi\):

SFT. For this estimator, we initialize \(v\) to some random or \(0\) value, choose an optimizer like Adam Kingma and Ba (2015), a learning rate (\(0.02\)), and some number of gradient steps or other early stopping criterion (300 steps), a batch size, and perform mini-batch stochastic gradient descent to optimize the following objective:

where \(p_\theta(y_c\mid x; v)\) is the conditional probability of the string \(y\), conditioned on \(x\), and where \(v\) is plugged into the forward pass of the network at the specified layer at all tokens. We can do this because our network is differentiable, as is the contribution of \(v\); that is, we can compute

APO-up. The direct preference optimization loss (DPO) (Rafailov et al., 2023) roughly teaches models to put more probability on chosen responses, and/or less probability on the rejected responses, relative to the initial model. We'll use a variant we'll call APO-up or just APO, inspired by D'Oosterlinck et al. (2025), which modifies DPO by adding a term to encourage the chosen response's probability to go up (since DPO can be minimized, for example, by just reducing the probability of the rejected response.) For APO, we again learn \(v\) by gradient descent, but for the following optimization problem:

Sampling

For evaluation, we apply the steering vector during sampling from models with temperature \(0.7\), applying the steering vector at every token. For DoM, we report both applying the steering vector as-estimated, as well as with a scalar multiple \(2\) to see if that increases steering efficacy (potentially at the cost of quality.) In the figures these are \(c=1\) and \(c=2\): during generation we add \(cv\) rather than \(v\), so the larger \(c\) is the setting meant to increase steering efficacy.

Models and hyperparameter selection

We evaluate our steering vector estimators for SmolLM2-360M (Allal et al., 2025), Qwen-2.5-1.5B, Qwen-2.5-3B, Qwen-2.5-7B, and Qwen-2.5-14B (Qwen Team, 2025). With an agent, I haggled around hyperparameters for a while with SmolLM2-360M, and fixed those for the other models. This included, e.g., trying various layers (landed on 40% of the way through the network), decoding multipliers for the steering vector (ended up going with multipliers of \(1\) and \(2\) for all methods), a learning rate of 0.02, 300 gradient steps, tried various \(\beta\) for DPO and landed on \(0.02\).

Results

SFT and APO beat the difference-of-means when Luna generates the data on average---always considerably stronger on quality, worse sentiment steering, roughly the same French steering, and then DoM totally fails to steer on Code style and sycophancy while the gradient-based methods perform well. When the models themselves generate the data, interestingly the gradient-base dmodels now perform better on sentiment, suggesting they were too focused on learning Luna's style on sentiment. On French they outperform DoM on quality and underperform slightly on steering efficacy. For code style and sycophancy for model-generated data, it looks like overall none of the methods learned that much, though the gradient-based methods may slightly overperform on steering efficacy.

Results for each task separately, averaged across models. Our first results use GPT-6-Luna as the generator of the training data. For French, DoM has the highest steering efficacy, at a substantial cost to overall quality. One critical takeaway is that DoM (either method) basically doesn't work at all for our 3-way if/then codestyle dataset. One might look at that and say well that makes sense; why would a 3-way if-statement implemented in code be a direction in activation space? There's no such direction. but that would be incorrect! If we simply learn the steering vectors by gradient descent, then we get a very large increase code style matching. See Figure 1.

When GPT-6-Luna is used as the generator, two things at least are being learned by the network: the dataset concept distribution (French, sycophancy etc), as well as the style/substance of the much more powerful model. Since the gradient-based methods specifically attempt to recreate the training distribution, the Luna data may be what causes higher quality. To me, this is a desirable attribute in general, but we should try to control for that, so in later experiments, we'll use the models themselves to generate the training sets.

Results plotted as a function of model size, averaged across tasks. Across model sizes (Figure 2), we see that steering efficacy and the order of the quality of steering vector estimations are roughly flat, while model qualities increase. Not much to see here, but makes sense.

Results (dataset generated by the target model, prompted)

Now we turn to our results wherein the training sets were generated by the target models. That means we have 4 training sets, and each model's steering vector is estimated from its training set. So, we know the model can generate these texts with reasonable probability under a prompt, and we've removed the confounder about the much more powerful teacher model.

Results for each task separately, averaged across models. Sentiment works better than before for our gradient-based estimators, and none of the model-generated data seems to teach much about code style or sycophancy, which is interesting. But either way, the gradient-based estimators are better than the difference-of-means estimators. See Figure 3.

Results plotted as a function of model size, averaged across tasks. The same holds for the results by model size (Figure 4). For the model-generated training datasets, the gradient-based estimators are better for most model sizes, and steering efficacy is roughly flat as quality increases with model size.

In a world with many unrelated steering vectors

So far, we've tackled the empirical question of what might be gained in terms of steering quality from considering different steering vector estimators. But there's also a scientific question here about the nature of claims like there is a 1-dimensional manifold that mediates the \(X\) concept. This is an existence claim, but it feels a bit like it's claiming uniqueness too, since if there are 1,000 independent 1-dimensional manifolds that all mediate \(X\), then it sort of seems like discovering the 1 isn't really the key insight we want.

Through gradient descent and through difference-of-means, we now have four or so vectors (two different DoM estimators, two different gradient-based estimators on likelihood-like losses.) They all, under our coarse evaluations, mediate something along the lines of the concept they were trained on. How similar are they?

See Figure 5 and Figure 6—between the two DoM estimators, kinda similar (\(0.28\) and \(0.25\) cosine sims for vectors from luna-generated and model-generated datasets). Between SFT and APO, vectors are pretty similar: \(0.55\) and \(0.72\) cosine sims on average. Across styles – similarity between a DoM vector and a gradient-estimated vector, almost orthogonal: \(0.05\) and \(0.06\) cosine similarities. This is compared to \(0.02\) and \(0.15\) cosine sims between the same DoM method across tasks, and \(0.14\) and \(0.14\) cosine sims across tasks for gradient-based estimators.

This isn't a massive surprise; models have a huge amount of redundancy. However, it's worth keeping in mind that when someone says they found the \(X\) vector, it's not clear that they found the best, or that a better-mediating vector would even look anything like the one they found. These networks are exceptionally expressive. There's a huge amount of nonlinear computation that models use to contextualize these steering vectors. To sheepishly cite my own work, How the Residual Stream is (Not) Linear (Hewitt, 2026).

Conclusion

So, why steer by difference-of-means? It's simple, and it often works. But when we want to really push the limits of our ability to discover simply-represented structure in networks, we can do much better. I think in general interpretability for safety has a lot to be learned from the tools that are just commonplace in capabilities research and deep learning more broadly.

Notes

1 Classically, what I'm calling the form is called the model. In this setting, this has confusing overlap, since we usually call the language model being interpreted the model. So we'll go with form.

References