Demos
The Mechanistic Sandbox
Welcome to the sandbox! The demos below run 100% locally in your browser using WebAssembly and Transformers.js.
(Note: The first time you interact with them, they will take a few seconds to download the neural networks into your browser’s cache. Subsequent loads will be instant!)
1. Tokenization
Before an LLM can understand language or pay attention to it, it has to chop words up into subword “tokens.” You can visualize exactly how models like GPT-4 slice up rare words (like indubitably) using OpenAI’s official Tokenizer Sandbox.
2. The Semantic Brain (Embeddings)
How does a neural network know that “The monarch rested” is similar to “The king sat” even though they share no words?
We use a feature extraction model (all-MiniLM-L6-v2) to turn text into a 384-dimensional dense vector space, and then calculate the Cosine Similarity between them! Try typing two sentences below to see their mathematical similarity.
3. The Live Attention Visualizer
Type a sentence below to generate an interactive map! Hover over any word in the sentence below to see what it is paying attention to, OR look at the full N × N Matrix grid beneath it.
[!NOTE] A Note on Approximation: Standard WebAssembly models physically delete their internal attention tensors to save space. To simulate the “attention” map live in your browser, this visualizer actually computes the Token-to-Token Semantic Similarity across the final layer of the network. It tells us how much the meaning of one token was blended with the meaning of another token, which visually and functionally serves as a beautiful approximation of self-attention!
4. The Canned Attention Visualizer
For educational purposes, here is a hard-coded visualisation of a pre-computed attention matrix extracted from a real Transformer model running offline. Hover over any token to see which other tokens it “attends” to!
Hover over a word to visualize its attention weights.