Everything so far in this course has assumed the input is text. Lesson 5 covered how text becomes a sequence of integers a transformer can process. Images, audio, and generated pixels are not sequences of discrete symbols to begin with. Making a transformer work with them at all requires a specific, separate translation step for each modality, and the four translation techniques in this lesson are what make that possible.
A transformer only ever consumes sequences of vectors. Every modality this lesson covers, images, speech, generated audio, generated pixels, gets forced into that same shape through a different, purpose-built mechanism, and the mechanism chosen determines what the resulting system can and cannot do well.
This lesson covers spatial patch tokenization for turning images into transformer input, contrastive alignment for connecting images and text in one shared space, the two different ways audio gets processed for transcription versus generation, and the iterative denoising process behind modern image generation, with a full worked numeric example for each mechanism where the source research provides one.
Section 01Turning an image into a sequence: the Vision Transformer
A transformer, as covered in Lesson 4, expects a 1D sequence of vectors as input. An image is a 2D grid of pixels. Dosovitskiy et al. (2020), in “An Image Is Worth 16x16 Words: Transformers for Image Recognition at Scale,” introduced the Vision Transformer (ViT), which resolves this mismatch without using any convolutional operations at all.
The mechanism, in exact form
An input image is a tensor X ∈ ℝH×W×C, where H is height, W is width, and C is the channel count, C = 3 for standard RGB images. Instead of feeding the raw pixel grid into the model, ViT partitions the image into a sequence of non-overlapping square patches, each of resolution P × P. The number of resulting patches, N, which becomes the sequence length the transformer actually processes, is:
Each 2D patch, of shape P × P × C, gets flattened into a single 1D vector containing P² · C numbers. Because that flattened vector’s length depends on the patch resolution and channel count rather than the transformer’s actual working dimension, a learned linear projection matrix E maps every flattened patch vector into the model’s hidden embedding dimension D.
A worked example: ViT-B/16 on a 224×224 image
Take the baseline ViT-B/16 configuration, processing a standard 224 × 224 pixel RGB image.
Sequence length: N = (224 × 224) / (16 × 16) = 50,176 / 256 = 196 patches.
Flattened patch dimension: P² · C = 16 × 16 × 3 = 768 values per patch.
Linear projection: the projection matrix E ∈ ℝ768×768 maps each 768-dimensional flattened patch into the model’s hidden dimension, D = 768.
Following the same pattern BERT uses for its classification token, a learnable class token vector xclass ∈ ℝD is prepended to the sequence of 196 projected patch vectors. To restore the spatial layout that flattening destroyed, learnable 1D position embeddings Epos ∈ ℝ^((N+1)×D) are added element-wise to the full sequence:
The resulting input tensor z₀ ∈ ℝ197×768, 196 patch tokens plus one class token, each 768-dimensional, is exactly the shape a standard transformer encoder expects, and it gets processed through ordinary multi-head self-attention and feedforward layers from that point on, no different from how a sequence of text tokens would be.
Why ViT needs much more data than a CNN
Convolutional neural networks have hardcoded inductive biases built into their architecture: translation equivariance, meaning a shifted version of a pattern is still recognized as the same pattern, and locality, meaning early layers only look at small neighboring pixel regions. A Vision Transformer has almost none of this built in. Self-attention connects every patch to every other patch, regardless of spatial distance, from the very first layer, and it has to learn any useful sense of spatial structure entirely from the position embeddings and the training data itself, rather than having that structure assumed by the architecture.
This has a direct, measured consequence. Trained on a mid-sized dataset like ImageNet-1k without heavy regularization, ViT underperforms CNNs of equivalent parameter count, because it lacks CNNs’ structural head start. But at large enough scale, pretrained on ImageNet-21k (14 million images) or the proprietary JFT-300M dataset (300 million images), that gap reverses. A pretrained ViT reaches 88.55% top-1 accuracy on ImageNet, while needing 2 to 4 times less compute during pretraining than a comparable convolutional baseline like EfficientNet. Scale substitutes for the structural assumptions a CNN gets for free.
| Attribute | Convolutional networks (CNNs) | Vision Transformers (ViT) |
|---|---|---|
| Input format | Continuous 2D pixel grid | Sequence of flattened 2D patches |
| Hardcoded inductive bias | High (translation equivariance, locality) | Low (global self-attention, learned positions) |
| Spatial feature extraction | Local sliding convolutional kernels | Global multi-head attention across all patches |
| Small-data efficiency | High; generalizes well on smaller datasets | Low; prone to overfitting without large scale |
| Compute scaling | Performance saturates earlier at extreme scale | Scales predictably with more data and compute |
Section 02Connecting images and text: CLIP’s shared embedding space
Turning an image into patch tokens is not the same as connecting that image to the meaning of a text description. Radford et al. (2021), in “Learning Transferable Visual Models From Natural Language Supervision,” introduced CLIP (Contrastive Language-Image Pre-training) to solve exactly that problem: mapping images and text into one shared vector space where semantic closeness in either modality corresponds to spatial closeness in the same geometry covered in Lesson 7.
Two encoders, one shared space
CLIP uses two separate encoders: a vision encoder EI (either a ResNet or a Vision Transformer, using the patch tokenization from Section 1) and a text encoder ET (a causal transformer, the same architecture family covered in Lesson 4).
Given a mini-batch of N image-text pairs {(x₁, y₁), (x₂, y₂), ..., (xN, yN)}, the vision encoder produces vᵢ = EI(xᵢ) ∈ ℝdv, and the text encoder produces tⱼ = ET(yⱼ) ∈ ℝdt. Two learned linear projection matrices, WI ∈ ℝde×dv and WT ∈ ℝde×dt, map both representations into the same shared embedding dimension de. The projected vectors are then L2-normalized to unit length, exactly the normalization step from Lesson 7 that makes cosine similarity reduce to a simple dot product:
The contrastive training objective
CLIP’s training goal is to make the N correctly-matched pairs (v̂ᵢ, t̂ᵢ) have high cosine similarity, while making the N² − N mismatched cross-pairs (v̂ᵢ, t̂ⱼ)_{i≠j} have low similarity. Every image and text embedding in the batch gets compared against every other one, producing a full pairwise similarity matrix S ∈ ℝN×N:
τ is a learnable temperature parameter that scales how sharply the similarity scores separate before softmax is applied. The full training loss is a symmetric cross-entropy, averaging the loss from matching each image to its correct text and matching each text to its correct image:
Each of these two directional losses is exactly the softmax cross-entropy loss from Lesson 2, applied here to a batch of N candidates instead of a fixed vocabulary, treating the correctly matched pair as the single right answer among N choices.
Once trained this way, CLIP supports zero-shot image classification with no task-specific fine-tuning at all. Class names get wrapped into a natural-language prompt template, such as “a photo of a {label},” passed through the text encoder to get a normalized text embedding, and then compared by dot product against a normalized image embedding. Whichever class’s text embedding scores highest becomes the model’s prediction, with no labeled training examples for that specific classification task ever required.
Section 03Two different jobs for audio: transcription vs. generation
Audio is a continuous, high-frequency 1D waveform, typically sampled at 16,000 to 48,000 samples per second. Multimodal systems handle audio with two structurally different techniques, depending on whether the goal is transcribing speech to text or generating new audio.
Whisper: continuous features for transcription
Radford et al. (2022/2023), in “Robust Speech Recognition via Large-Scale Weak Supervision,” introduced Whisper, an encoder-decoder transformer trained on 680,000 hours of weakly supervised audio for multilingual speech recognition, translation, and language identification.
Whisper’s pipeline runs in three stages.
- Preprocessing. Audio gets resampled to a standard 16,000 Hz and split into fixed 30-second windows. An 80-channel log-magnitude Mel-spectrogram is computed using a 25-millisecond analysis window with a 10-millisecond hop between windows, producing an 80 × 3,000 feature matrix for each 30-second chunk: 80 frequency bins across 3,000 time frames.
- Convolutional audio stem. That 80 × 3,000 spectrogram passes through two 1D convolutional layers, each with a filter width of 3 and a stride of 2, which together downsample the time dimension by a factor of 4. The result is a sequence of 750 feature vectors per 30-second window, an effective frame rate of one vector every 40 milliseconds, and this is what actually feeds into the transformer encoder.
- Multitask decoder. The transformer decoder generates output autoregressively, the same left-to-right generation mechanism from Lesson 4, predicting a mix of text tokens and special control tokens such as
<|START|>,<|LANG_ID|>,<|TRANSCRIBE|>,<|TRANSLATE|>, and<|NOTIMESTAMPS|>. Timestamps get quantized to 20-millisecond intervals and represented as their own distinct vocabulary tokens, which lets the model output text that is aligned in time to the original audio, not just a flat transcript.
EnCodec: discrete codes for generation
Whisper’s continuous spectrogram works well for transcription, but a causal autoregressive decoder, the mechanism used to generate new content, needs to predict from a fixed, finite vocabulary at each step, exactly like the token-by-token text generation in Lesson 6. A raw continuous audio signal has no such fixed vocabulary to sample from. Défossez et al. (2022), in “High Fidelity Neural Audio Compression,” introduced EnCodec to solve exactly this mismatch for generative audio.
EnCodec first compresses raw audio (24 kHz mono or 48 kHz stereo) into a continuous latent sequence z ∈ ℝT×D at a reduced frame rate. It then converts that continuous latent into a discrete sequence using Residual Vector Quantization (RVQ) across K cascaded codebooks (Q₁, Q₂, ..., QK):
The first codebook, Q₁, quantizes the continuous latent z directly, capturing the primary, low-frequency structure of the sound. Each subsequent codebook doesn’t quantize the original signal again. It quantizes whatever residual error, r, was left over after the previous codebook’s approximation, progressively capturing finer and finer high-frequency detail that the earlier stages missed. The end result is K parallel streams of discrete codebook indices, one integer per stream per time step, which is exactly the shape a causal transformer decoder needs to generate new audio one discrete token at a time, the same way it would generate text.
Section 04Generating images: denoising diffusion
Text and speech transcription are both generated autoregressively, one discrete token at a time. High-fidelity image generation predominantly works differently, using an iterative denoising process instead. Ho et al. (2020) formalized this in “Denoising Diffusion Probabilistic Models” (DDPM), building on principles from nonequilibrium thermodynamics.
The forward process: destroying an image with noise
The forward diffusion process takes a clean image x₀ and gradually adds Gaussian noise over T discrete timesteps, following a fixed variance schedule β₁, β₂, ..., βT, where each βₜ ∈ (0, 1):
Defining αₜ = 1 − βₜ and ᾱₜ = Πₛ₌₁ᵗ αₛ, the reparameterization trick gives a closed-form way to jump directly to any timestep t, without stepping through the full chain one step at a time:
At the final step, T (typically set to 1,000), the noise schedule has accumulated enough that xT approximates pure isotropic Gaussian noise, N(0, I). By the end of the forward process, essentially all of the original image’s structure has been destroyed.
The reverse process: learning to undo the noise
Generation runs this process backward: starting from pure Gaussian noise xT ~ N(0, I), a learned reverse process removes noise step by step to reconstruct a clean image. Each reverse transition is parameterized by a neural network:
What the network is actually trained to predict
Ho et al.’s key finding was that training the network εθ(xₜ, t) to predict the noise vector ε that was added at step t, rather than trying to directly predict the clean image itself, produces better sample quality. The simplified training objective is a mean-squared error between the true added noise and the network’s prediction of it:
t is sampled uniformly from {1, ..., T}, x₀ is a real training image, and ε ~ N(0, I) is the actual noise that was added. This is structurally the same mean-squared-error loss covered in Lesson 2, applied here to a noise vector instead of a scalar prediction.
Diffusion vs. autoregressive generation, side by side
| Property | Autoregressive text generation | Denoising diffusion image generation |
|---|---|---|
| Generation principle | Causal, sequential prediction of discrete tokens | Iterative, stochastic denoising of a continuous tensor |
| Execution order | One-directional token accumulation | Stepwise refinement across the whole spatial grid at once |
| Output size during generation | Grows by one token per step | Constant spatial grid size throughout |
| What each step predicts | A discrete token index via softmax | A continuous noise vector, matching the image’s shape |
| Generation latency | Proportional to number of tokens generated | Proportional to the total number of denoising timesteps, T |
Diffusion Transformers: replacing the UNet with a transformer
Early DDPM implementations used a convolutional UNet as the network doing the denoising. Peebles and Xie (2022), in “Scalable Diffusion Models with Transformers,” introduced the Diffusion Transformer (DiT), replacing that convolutional backbone with a pure transformer operating on spatial patches, using the exact same patch-tokenization idea from Section 1, but applied to the latent representation produced by a separate Variational Autoencoder rather than to raw pixels. DiT conditions its transformer blocks on both the current diffusion timestep and any text prompt embedding, using adaptive layer normalization. Like the transformer language models covered earlier in this course, DiT’s image generation quality scales predictably with more parameters and more training compute.
Section 05Paper-grounded mechanisms vs. commercial products
Every mechanism covered in this lesson, patch tokenization, contrastive alignment, spectrogram-based transcription, residual vector quantization, and denoising diffusion, is published, peer-reviewed research with a fully specified mathematical formulation. The specific commercial systems that use these mechanisms are a different matter. Their exact internal configurations, patch sizes, hidden dimensions, layer counts, and codebook structures, are not publicly disclosed, and treating any claim about their internals as verified fact would be a mistake.
| System component | Durable, paper-grounded mechanism | Commercial example | Verification status |
|---|---|---|---|
| Visual feature extraction | Vision Transformer (Dosovitskiy et al., 2020) | Google Gemini 1.5 (Feb 2024) | UNVERIFIED internal configuration; grounded only in the general patch-tokenization concept |
| Native multimodal processing | Multimodal sequence transformers | OpenAI GPT-4o (May 2024) | UNVERIFIED parameter weights and codebooks |
| Text-guided image synthesis | CLIP contrastive learning (Radford et al., 2021) | OpenAI DALL-E 3 (Oct 2023) | UNVERIFIED production pipeline; believed to combine re-captioning with a latent diffusion backbone |
| Video generation | Diffusion Transformers (Peebles and Xie, 2022) | OpenAI Sora (Feb 2024) | UNVERIFIED parameter weights; extends 2D patches into 3D spacetime patches |
| Iterative latent denoising | DDPM framework (Ho et al., 2020) | Midjourney v6 (Dec 2023) | UNVERIFIED proprietary implementation |
(Commercial product names, release dates, and architectural claims are a dated snapshot and should be treated as such; the underlying mathematical mechanisms are the durable part of this table.)
Two specific commercial systems are worth naming as concrete illustrations of how these mechanisms combine in practice, while keeping the same caveat in mind. OpenAI’s Sora is reported to adapt 2D spatial patch extraction into 3D spacetime patch tokenization, splitting a video across both space and time into latent blocks processed by a Diffusion Transformer backbone. Google’s Gemini 1.5 and OpenAI’s GPT-4o are reported to process interleaved text tokens, image patches, and continuous audio representations inside one unified context window. Neither claim can be verified against a primary published architecture paper, since neither system’s internals are open.
Conclusion
Every mechanism in this lesson exists to solve the same underlying problem from a different angle: a transformer only knows how to consume a sequence of vectors, and none of these input types start out in that shape. Patch tokenization reshapes a 2D image grid into a 1D sequence, with a linear projection and position embeddings restoring what flattening destroyed. CLIP’s contrastive loss does not tokenize anything new, it aligns two already-tokenized modalities into one shared geometric space, so that a dot product can stand in for semantic similarity across images and text. Whisper and EnCodec solve two different audio problems with two different representations, continuous spectrograms for transcription, discrete residual codes for generation, because a transcription decoder needs to read structure while a generation decoder needs to sample from a finite vocabulary. And diffusion abandons the token-by-token generation pattern entirely, refining an entire noisy image all at once instead, because natural images do not decompose cleanly into a left-to-right sequence the way text does.
The next lesson turns to a failure mode that touches every one of these systems in different ways: what happens when a model, text or otherwise, produces a fluent, confident output that is simply wrong, and why the next-token training objective from Lesson 6 does not, on its own, select for truthfulness.