Tips
Halt! Who goes there?
To access the Tips, you must complete this quote:
Welcome to my Digital Garden! Here is a collection of living documents, tips, and notes I iterate on.
Because Large Language Models use subword tokenization (like BPE), a word like indubitably might be split into in, du, bit, ably. If you try to visualize the raw attention matrix, you get a massive Token × Token grid that looks like a noisy, unreadable mess to human eyes.
To build a human-readable Word-Level Attention Map, we have to mathematically aggregate those token weights.
If Word A is made of 3 tokens, and Word B is made of 2 tokens, how much does Word A attend to Word B? We calculate the “attention from Word A to Word B” by taking the mathematical average of the attention weights going from all 3 sub-tokens of A to all 2 sub-tokens of B.
It’s a mathematical compromise, but it collapses the massive token grid into a clean Word × Word 2D matrix that we can actually interpret.
The discovery of the “Spiderman Neuron” in AI models is actually a direct parallel to a famous discovery in human neuroscience: the Halle Berry Neuron.
In 2005, researchers studying epileptic patients with implanted brain electrodes discovered a single neuron in a patient’s medial temporal lobe that fired exclusively in response to Halle Berry.
It fired when the patient saw:
- A photo of Halle Berry.
- A drawing of Halle Berry.
- Halle Berry dressed as Catwoman.
- The written text “Halle Berry”.
It didn’t fire for other actresses, or even for pictures of Halle Berry that didn’t look like her. It was a concept cell: a neuron that had learned the abstract identity of a specific person, completely invariant to the visual medium.
In 2021, OpenAI researchers were digging into the internal representations of their CLIP model (a multimodal network trained to link images and text), looking for Concept Cells.
When they mapped the activations, they found a literal “Spiderman Neuron” (Neuron 53 in one of the final ResNet layers). This single neuron would fire intensely when the model was fed:
- A photo of someone in a Spiderman costume.
- A comic book illustration of Spiderman.
- The literal text word “Spider-Man”.
- A completely abstract, black-and-white sketch of a spider web.
It proved that deep inside the matrix, the model had dedicated a specific piece of its architecture to the abstract, high-level concept of Spiderman across multiple dimensions of media.