The words are the shadow

One object, several shadows

In 2009, Niloy J. Mitra and Mark Pauly published Shadow Art. Their algorithm searches for a single sculpture that can cast several requested silhouettes. On its own, the object can look like a very suspicious piece of plastic.

Start with this object. From one light direction, its shadow is a bull. From another, a winged creature. From a third, an eagle. The object stays put. The image on the wall changes.

This is computational shadow art. In the 2009 work, you specify the silhouettes and projection directions, and an algorithm searches for a compatible sculpture. Some requests conflict: the material needed for one shadow gets in the way of another. The method adjusts the targets to make the geometry possible.

Apparently even a piece of plastic has to negotiate conflicting requirements.

Then there is Mojoptix’s digital sundial. As the Sun moves, carefully arranged channels let light through to form changing digits inside its shadow. No battery. No screen. A clock whose display driver is the rotation of the Earth.

That gives me a useful way to picture a representation: the thing carrying the information can be richer than any one image we read from it. A readable shadow needn’t resemble the sculpture that produced it.

Now imagine the sculpture changes too

Now take that metaphor and add motion. A language model doesn’t just cast a static shadow. The thing we’re picturing as the sculpture changes as information passes through the network.

In DeepSeek-R1-Distill-Qwen-7B—the model I was trying to adapt—28 transformer blocks run in sequence. A vector is a point described by coordinates. During prompt processing, the model carries one such state for each token position. Tokens can be pieces of words.

Its published configuration gives each state 3,584 coordinates. A 512-token prompt is a constellation of 512 points, each described by 3,584 numbers. “High-dimensional” doesn’t mean a mystical realm behind the universe. It means that many coordinates. The sculpture makes the idea visible; the model isn’t storing a solid object.

The learned query, key, and value matrices take different linear views of the representations. “Move the light” is a useful starting picture, but these transformations can scale and mix coordinates as well as project them. Attention then calculates input-dependent weights and mixes information from the available token positions. The original transformer paper gives the mechanism.

The MLP applies a nonlinear transformation at each position, and residual connections add the resulting updates back. The representation itself changes. The next block works with the updated state. The transformer-circuits framework describes this shared stream as a communication channel between components.

For “bank,” the words “river” and “mortgage” provide different context. The model can use that context to change what it carries forward. We don’t need to invent a dedicated “riverbank removal department” on layer 12 to explain the idea.

The model can learn a procedure

This is the sense in which “soft execution engine” becomes a useful picture. Attention can weight and combine several pieces of information. Successive blocks can use intermediate results. Learned weights can implement a procedure that runs on the current context, even while those weights stay fixed.

The layer wiring of this dense model is specified in advance. What changes with the input is the state and the attention weights that determine how information is mixed. The blueprint stays; the traffic changes. That gives us a useful picture of input-dependent computation through learned circuitry, without assuming the model compiles a fresh symbolic program for every reply.

One concrete example is the induction circuit studied in small, attention-only transformers. Given a pattern like A B … A, it helps predict B: find an earlier matching context, then copy what followed it. Researchers identified how attention heads in successive layers cooperate to do that. It is a learned copying procedure, and its evidence is strongest in those small models.

Another is Othello-GPT. Researchers trained a transformer on sequences of board-game moves, without giving it the board or explicit rules as inputs. They found evidence for an internal representation of the board state. Intervening on that representation changed its move predictions. Predicting the next move had led to something useful for prediction: tracking the situation those moves created.

The training target didn’t say, “Please build a board in there.” Apparently “guess the next thing” can turn into a more involved job description.

There is a useful resemblance to thinking here: carrying a state forward and using it to decide what comes next. These studies establish particular representations and computations in particular models. They don’t establish that a transformer has a human mind, or identify the procedure behind any particular reply from the model I used.

One line from my conversation with Gemini captured why I find this picture so compelling:

Human language is just the serialized, lossy output format of a complex, high-dimensional computation graph executing inside the network's layers.

I read that as a metaphor for the model’s output: rich internal computations become a sequence we can read. Those words can then become part of the next computation, which is why the scratchpad matters too.

That also makes language part of the computation. Once the model writes an intermediate step, later tokens can use it as context. In Show Your Work, researchers trained models to write scratchpads and found that this helped with multi-step computations. Language can be both a readout and a scratchpad. The model needn’t finish the whole calculation before it starts writing.

The scratchpad still needs checking. Research on unfaithful explanations shows that plausible reasoning text needn’t report every influence on an answer. My saved replies let me inspect the behavior; they don’t expose a complete internal execution trace.

How a state becomes the next word

At the output, the shadow picture becomes useful again. After final normalization, the state at the current output position is compared with a learned vector for each vocabulary entry. Each dot product gives one score; a matrix multiplication collects those scores together. “Dot products” and “a linear projection to vocabulary scores” describe the same readout. This isn’t an orthogonal projection onto a literal wall.

The published configuration specifies 152,064 output scores from a 3,584-value state. More scores don’t create more independent dimensions. Think of many learned measuring directions looking at the same state together. They aren’t one tidy axis per concept, and the vocabulary includes word fragments and punctuation.

Softmax turns the scores into probabilities, then the decoding rule selects a token. The task is predicting a continuation, not finding the word most similar in meaning to the prompt. After “The capital of France is,” the useful continuation would be “Paris,” not another word resembling “France.”

The reverse problem: choose the shadows

Shadow Art Revisited, presented at WACV 2022, uses differentiable rendering: render a candidate’s shadows, compare them with the requested images, and use gradients to adjust its geometry. Repeat. Its examples include letters, animal silhouettes, faces, and a car constrained by several views.

Neural Shadow Art, first posted in 2024 and revised for Pacific Graphics 2025, extends this using a neural representation of occupied space. It can optimize light directions and screen orientations alongside the sculpture. The paper includes a fabricated example.

These papers design physical geometry. They make the analogy visible; they don’t establish how my model represents sarcasm.

The language-model geometry has its own public research. Park, Choe, and Veitch study how concept directions relate to readouts and interventions. Toy Models of Superposition demonstrates how sparse features can share fewer dimensions in simplified networks. There needn’t be a separate coordinate labeled “sarcasm.” Apparently the concepts have to share office space too.

I now have three things to keep separate: the states carrying context, the learned rules that update them, and the vocabulary scores we read out. The computation graph names the operations and their dependencies. Its wiring can stay fixed while the states and attention patterns change from one input to another.

That distinction also matters when I try to change a model. A different prompt changes its context. Activation steering changes a passing state. Training an adapter changes some of the learned rules. My attempt to teach a model sarcasm tested those interventions; the sculpture is a way to think about them, not evidence that I found a sarcasm mechanism.

I still want to know which computations make a tone survive a change of subject. At least now I have a better question than “which layer do I put the sarcasm in?”