Author(s): Latent AI Labs

Originally published on Towards AI.

How AI Generates Video: Inside Sora and Wan2.1 on a Real GPU

Watch the full 22-minute animated video companion on YouTube here:

TL;DR — What You Will Learn in This Deep Dive

• Why 2D Image Diffusion Fails at Video: Generating frames one by one in single-frame (T = 1) 2D mode with independent Gaussian noise collapses frame-to-frame consistency to 10.12 dB PSNR (a 57.15 mean pixel jump on a 0 to 255 scale) and uncorrelated motion (-0.0607 optical flow cosine). A 3D Diffusion Transformer (DiT) generating 17 frames jointly raises temporal consistency to 27.74 dB PSNR (4.11 mean pixel diff) and +0.5422 optical flow cosine.

• The 1.046 Quadrillion Attention Wall: A 5-second video at 832×480 resolution and 16 fps contains 81 frames (32,348,160 RGB pixels). Full self-attention over raw pixels requires 1.046 quadrillion (1.0464 x 10¹⁵) pairwise comparisons — demanding 2,093 TB of VRAM per attention head in fp16.

• 3D Causal VAE Compression: Alibaba’s AutoencoderKLWan (126.89M parameters) compresses height and width by 8x and time by 4x after keeping Frame 0 intact (T' = 1 + (T - 1) / 4). An 81-frame tensor ([1, 3, 81, 480, 832], 97.04M floats) shrinks into a [1, 16, 21, 60, 104] latent volume (2.10M floats — a 46.29x float compression and 256:1 pixel-to-position reduction per interior 4x8x8 block) with 0.000000 past causal leakage.

• Spacetime Tubes and 3D RoPE: Grouping [16, 1, 2, 2] latent blocks into 1,536-D spacetime tube tokens reduces an 81-frame clip to 32,760 tokens (1.073 billion attention pairs, 2.15 GB in fp16 — a 975,015x compute and memory reduction). Each 128-D attention head splits its rotary position embedding (axes_dim = [44, 42, 42]) into 22 Time, 21 Height, and 21 Width frequency pairs.

• Simultaneous 4D Flow Matching & 90.4% Cross-Time Attention: Unlike autoregressive LLMs that generate token by token, a Video DiT denoises all frames simultaneously over 30 Rectified Flow steps with flow_shift = 3.0. Probing Layer 21 Head 7 on our GPU reveals that a token on a moving car's wheel spends only 9.63% of its attention in its own time slice and routes 90.37% across past and future slices (40.44% in t' = 3 alone) to track motion.

• Why Video AI Still Breaks Physics: Because 3D RoPE only encodes [t', h, w] screen coordinates with no depth axis Z, when a stone pillar occludes a moving object in slices t' = 2 and t' = 3, the stone texture overwrites the object's 1,536-D tube features — collapsing feature cosine similarity from 0.9760 to 0.2792 (a 71.4% signal wipeout) and causing a yellow tennis ball to stall in front of the pillar and distort instead of passing behind it.

1. The Temporal Flicker Problem: Why 2D Diffusion Cannot Make Video

Direct Answer: A 2D image diffusion model cannot generate coherent video because sampling each frame independently from random Gaussian noise gives the network zero shared memory of camera angle, cliff geometry, or object position across time. On an NVIDIA RTX PRO 6000 Blackwell GPU, generating single-frame (T = 1) outputs across consecutive random seeds yields a chaotic 10.12 dB frame-to-frame PSNR and a -0.0607 optical flow cosine similarity, whereas generating all 17 frames jointly inside a 3D Diffusion Transformer (Wan2.1-T2V-1.3B) yields 27.74 dB frame-to-frame PSNR and +0.5422 optical flow alignment.

In Episode 5 of this series, we saw how an image diffusion model turns a 2D grid of Gaussian static into a single crisp photograph by following a learned velocity vector field from noise (sigma = 1.0) to clean data (sigma = 0.0).

What happens if we try to make a video of a red sports car driving along a winding coastal cliff highway at golden sunset with crashing ocean waves by running single-frame (T = 1) 2D generation frame after frame?

Even if we feed the exact same text prompt into every single frame, each frame starts from its own independent draw of random Gaussian noise. Because Frame 2 cannot see the latent grid of Frame 1:

• Frame 1 places the red sports car in the foreground heading away from the camera with rocky cliffs on the right and golden sunbeams in the center.
• Frame 2 invents a completely different cliff overhang, flips the highway curvature, and moves the car fifty yards down the road.
• Frame 3 changes the camera elevation, ocean wave foam pattern, and car body proportions entirely.

When we play those frames back sequentially, our eyes are hit by violent visual static. We measured this gap directly on an NVIDIA RTX PRO 6000 Blackwell Server Edition GPU (94.97 GB VRAM) using Alibaba’s open-weight Wan2.1-T2V-1.3B architecture (WanTransformer3DModel, 1.419 billion parameters across 30 transformer blocks):

• Independent Single-Frame 2D Generation (5 consecutive T = 1 runs at 832×480, seeds 42..46): Mean frame-to-frame Peak Signal-to-Noise Ratio (PSNR) collapses to 10.12 dB. The average pixel jumps by 57.15 intensity levels (on a 0 to 255 RGB scale) between adjacent frames, and Farneback optical flow vectors point in random directions with a temporal cosine similarity of -0.0607.
• Simultaneous 3D Video DiT Generation (T = 17 frames generated jointly at 832×480): Mean frame-to-frame PSNR jumps by +17.62 dB to 27.74 dB. The mean pixel difference between consecutive frames drops 14x to just 4.11 intensity levels, and optical flow vectors lock onto a smooth trajectory with a +0.5422 temporal cosine similarity.

To eliminate temporal flicker, a video generator must stop treating a video as a slideshow of separate 2D photographs. Instead, it must treat the entire video clip — from Frame 0 to Frame 80 — as one unified 4D block of spacetime ([Channels, Time, Height, Width]) and denoise every frame simultaneously inside the same neural network forward pass.

2. The 4D Data Explosion and the 1.046 Quadrillion Attention Wall

Direct Answer: Adding a time dimension causes a quadratic explosion in self-attention compute that makes raw-pixel video transformers physically impossible to run. A single 5-second video clip at 832×480 resolution and 16 fps contains 81 frames or 32,348,160 RGB pixels, meaning a single layer of full self-attention would have to compute 1.046 quadrillion (1.0464 x 10¹⁵) pairwise dot products and store 2,093 TB of fp16 attention weights per head.

Before we can connect all frames together with self-attention, we have to confront the brutal arithmetic of 4D video tensors.

Consider a standard 5-second video clip generated by Wan2.1 at widescreen 832×480 resolution and 16 frames per second:

• Spatial pixels per frame: 832 * 480 = 399,360 pixels (1,198,080 RGB floating-point values).
• Total frames in 5 seconds: 81 frames (T = 81).
• Total pixel positions in the 5-second clip: 81 * 832 * 480 = 32,348,160 pixels.
• Total raw floating-point numbers ([1, 3, 81, 480, 832]): 97,044,480 floats.

If we tried to feed each pixel position (N = 32,348,160) directly into a Transformer self-attention layer, every pixel in every frame would have to compute a query-key dot product against every other pixel across all 81 frames:

• Raw Pixel Attention Pairs (N^2): 32,348,160 * 32,348,160 = 1,046,403,455,385,600 (1.046 quadrillion pairs).
• VRAM Required for One Attention Matrix in fp16 (2 bytes per pair): 2,092,806,910,771,200 bytes, or 2,092.8 TB of GPU memory — for a single attention head in a single layer!

Even our 95 GB RTX PRO 6000 Blackwell GPU (or an 80 GB H100) would run out of memory by a factor of 22,000x to 26,000x on the very first matrix multiplication. How do modern video architectures like OpenAI’s Sora, Meta’s Movie Gen, Kling, Veo, and Alibaba’s Wan2.1 crush a 2,093 TB attention matrix down into something that runs in 11.67 seconds on a single GPU?

They use a two-stage compression pipeline:

1. Stage 1 — A 3D Causal Variational Autoencoder (AutoencoderKLWan): Compresses the raw RGB video across both space (8×8) and time (4x) into a compact 4D latent tensor.
2. Stage 2–3D Spacetime Tube Patchification: Bundles 1x2x2 blocks of latent cells into spacetime tube tokens, slashing the sequence length from 32,348,160 pixels down to 32,760 tokens and shrinking the attention matrix by 975,015.2x.

Let’s open the hood on both stages.

3. Inside the 3D Causal VAE: Crushing Time by 4x and Space by 8×8

Direct Answer: Unlike 2D image VAEs that compress each frame independently and cause temporal shimmer, a 3D Causal Video VAE (AutoencoderKLWan, 126.89M parameters) uses 3D convolutions with causal left-only temporal padding to compress height by 8x, width by 8x, and time by 4x (T' = 1 + (T - 1) / 4). It keeps Frame 0 as a solo anchor slice (44.53 dB reconstruction PSNR), groups subsequent frames into 4-frame temporal chunks (40.18 dB mean PSNR), and guarantees 0.000000 backward temporal leakage from future frames into past slices.

In Stable Diffusion XL or Flux, a 2D VAE compresses an image’s height and width by 8x each (turning an 8x8 patch of 64 pixels into a single latent vector), but it knows nothing about time. If you encode a video frame-by-frame with a 2D VAE, two fatal problems occur:

1. No Temporal Compression: An 81-frame video still leaves you with 81 full latent frames, keeping your Transformer sequence length 4x too long.
2. High-Frequency Temporal Shimmer: Tiny frame-to-frame quantization differences in the 2D encoder create shimmering textures when decoded back to RGB.

Modern video generators solve both problems with a 3D Causal Variational Autoencoder (AutoencoderKLWan, 126.89 million parameters in Wan2.1). Instead of sliding a flat 3x3 2D kernel over a single image, it slides a 3D volumetric kernel (3 x 3 x 3 across [Time, Height, Width]) through the video volume:

• Spatial Compression (8x Height, 8x Width): 480 / 8 = 60 latent rows, and 832 / 8 = 104 latent columns.
• Asymmetric Temporal Compression (T' = 1 + (T - 1) / 4): Notice why Wan2.1 always generates clips of 17 frames, 33 frames, or 81 frames (4k + 1). Frame 0 is never mixed with other frames — it is compressed spatially (8x8) into its own dedicated first latent slice t' = 0. Every subsequent block of 4 consecutive frames (Frames 1..4, Frames 5..8, Frames 9..12, Frames 13..16) is fused along the time axis by a 4x temporal stride into a single latent slice (t' = 1, t' = 2, t' = 3, t' = 4).
• 17-Frame Video Compression (Measured on GPU): A raw 17-frame tensor of shape [1, 3, 17, 480, 832] (20,367,360 floats) is compressed in 1,077.73 ms (encode + decode) into a 4D latent tensor of shape [1, 16, 5, 60, 104] (499,200 floats — a 40.80x total float reduction).
• 81-Frame Video Compression (Measured on GPU): A 5-second, 81-frame tensor of shape [1, 3, 81, 480, 832] (97,044,480 floats) shrinks into [1, 16, 21, 60, 104] (2,096,640 floats — a 46.29x total float reduction, and a 256:1 spatiotemporal pixel-to-cell compression ratio inside every interior 4 x 8 x 8 video block).

Why does the 3D VAE isolate Frame 0 in slice t' = 0 instead of grouping Frames 0..3 together? For two critical architectural reasons:

1. Ultra-Sharp First-Frame Anchor: On our GPU benchmark, reconstructing Frame 0 from solo slice t' = 0 achieves a near-lossless 44.53 dB PSNR, compared to 40.18 dB mean PSNR across interior frames 1..16 that share 4-frame slices (41.25 dB on Frame 1, 40.31 dB on Frame 8, 40.67 dB on Frame 16; range 38.66–41.76 dB).
2. Unified Image-to-Video (I2V) & Text-to-Video (T2V) Latent Space: Because t' = 0 depends only on Frame 0, an Image-to-Video model can encode a single user reference photograph directly into t' = 0 without needing three future frames that do not exist yet!

Why “Causal” 3D Convolutions Matter: The Zero-Leakage Proof

Why is the word Causal in WanCausalConv3d so important if the Diffusion Transformer is going to look at all frames simultaneously anyway?

In a standard PyTorch nn.Conv3d(kernel_size=(3, 3, 3), padding=(1, 1, 1)), the convolution window is symmetric in time: to encode frame t, the kernel reaches one step into the past (t - 1) and one step into the future (t + 1). If the VAE encoder peeks into future frames:

• You cannot encode or decode long videos in streaming chunks without boundary artifacts.
• Corrupting or changing the end of a video alters the latent representation of the beginning of the video.
• A single image (a 1-frame video) cannot be encoded cleanly with the same weights as a multi-frame video.

WanCausalConv3d fixes this by replacing symmetric temporal padding with left-only causal padding of size k_t - 1 = 2 along the time axis (replicating the first frame at the boundary) and zero padding (0) into the future.

We verified this on our GPU by running a Future Corruption Leakage Test (M2 in measure_video_gen.py): we encoded our clean 17-frame video ([1, 3, 17, 480, 832]), then created a corrupted copy where only future frames 13, 14, 15, and 16 (corresponding to the final latent slice t' = 4) were replaced with clamped Gaussian noise in [-1.0, 1.0], and measured the maximum absolute change in every latent slice t' = 0..4:

• Wan2.1 CausalConv3d Encoder (t' = 0, 1, 2, 3, 4 max abs diff): [0.0, 0.0, 0.0, 0.0, 9.3173]. Past slices t' = 0, 1, 2, 3 change by 0.000000 — absolute mathematical isolation from the future.
• Standard Symmetric Conv3d Baseline: Leaks 0.149431 of the future corruption backward into past slice t' = 3.

Here is a self-contained, runnable PyTorch implementation showing how CausalConv3d enforces zero backward leakage at the exact boundary slice (t' = 6) right before corrupted frames 13..16, compared to standard nn.Conv3d:

import torch
import torch.nn as nn
import torch.nn.functional as F

class CausalConv3d(nn.Module):
 """
 3D Convolution with strictly causal (past-only) temporal padding,
 matching AutoencoderKLWan's WanCausalConv3d design.
 """
 def __init__(self, in_channels: int, out_channels: int, kernel_size=(3, 3, 3), stride=(1, 1, 1)):
 super().__init__()
 self.kt, self.kh, self.kw = kernel_size
 self.conv = nn.Conv3d(
 in_channels, out_channels, kernel_size=kernel_size, stride=stride, padding=0
 ) def forward(self, x: torch.Tensor) -> torch.Tensor:
 # x shape: [B, C, T, H, W]
 # Pad spatial dims symmetrically (kh//2, kw//2), but pad Time ONLY on the left (kt - 1, 0)
 pad_h, pad_w = self.kh // 2, self.kw // 2
 pad_t_left = self.kt - 1
 # First replicate-pad the temporal start so Frame 0 doesn't see zero-padding boundary drops
 x_t_padded = F.pad(x, (0, 0, 0, 0, pad_t_left, 0), mode="replicate")
 # Then zero-pad spatial height and width
 x_padded = F.pad(x_t_padded, (pad_w, pad_w, pad_h, pad_h, 0, 0), mode="constant", value=0.0)
 return self.conv(x_padded)# Verify zero backward temporal leakage on a 17-frame toy video [1, 3, 17, 32, 32]
torch.manual_seed(42)
video_clean = torch.randn(1, 3, 17, 32, 32)
video_corrupt_future = video_clean.clone()
video_corrupt_future[:, :, 13:17, :, :] += 10.0 # Corrupt only future frames 13..16causal_down = CausalConv3d(3, 16, kernel_size=(3, 3, 3), stride=(2, 2, 2))
symmetric_down = nn.Conv3d(3, 16, kernel_size=(3, 3, 3), stride=(2, 2, 2), padding=(1, 1, 1))with torch.no_grad():
 diff_causal = (causal_down(video_clean) - causal_down(video_corrupt_future)).abs()
 diff_sym = (symmetric_down(video_clean) - symmetric_down(video_corrupt_future)).abs()print("CausalConv3d max diff in past slice t'=6 (before frame 13):", diff_causal[:, :, 6].max().item())
print("Symmetric Conv3d leakage in past slice t'=6 (before frame 13):", diff_sym[:, :, 6].max().item())

4. Spacetime Tubes and 3D Factorized RoPE: Giving Tokens a 3D GPS Address

Direct Answer: After the 3D VAE produces a [1, 16, T', 60, 104] latent volume, the Diffusion Transformer slices it into 1 x 2 x 2 spacetime tube patches (64 floats each) and projects every brick into a 1,536-dimensional token (7,800 tokens for 17 frames; 32,760 tokens for 81 frames). To tell the Transformer where each brick sits in 3D spacetime, 3D Factorized Rotary Position Embedding (3D RoPE) splits each 128-dimensional attention head into three orthogonal complex rotation blocks: 44 dimensions (22 frequency pairs) for Time t', 42 dimensions (21 pairs) for Height h, and 42 dimensions (21 pairs) for Width w.

Once our 17-frame video is inside the VAE latent space with shape [1, 16, 5, 60, 104], how does WanTransformer3DModel turn that 4D grid into a 1D sequence of Transformer tokens?

In a 2D Vision Transformer (ViT), we cut a 2D image into flat 2x2 squares. In a 3D Video Diffusion Transformer (introduced by Peebles & Xie in DiT and scaled by OpenAI's Sora as spacetime patches), we cut the 4D latent volume into 3D Spacetime Bricks using a 3D strided convolution with patch_size = (1, 2, 2) and kernel_size = (1, 2, 2):

• Dimensions of One Spacetime Brick: 1 latent time slice (t') x 2 latent rows (h) x 2 latent columns (w) across all 16 latent channels = 16 * 1 * 2 * 2 = 64 floating-point numbers.
• Physical Footprint in the Original Video: Because each interior latent slice already spans 4 RGB frames and each latent cell spans 8x8 RGB pixels, a single [1, 2, 2] interior spacetime brick represents a 4 frames x 16 pixels x 16 pixels tube of raw video (1,024 RGB pixels, or 256 pixels on solo Frame 0)!
• Linear Projection to 1,536-D: The patch embedding layer maps those 64 latent floats into a 1,536-dimensional vector (hidden_size = 1536).
• Token Grid for 17 Frames ([5, 60, 104] latent): [5 / 1, 60 / 2, 104 / 2] = [5, 30, 52] = 7,800 spacetime tokens (1,560 tokens per temporal slice).
• Token Grid for 81 Frames ([21, 60, 104] latent): [21 / 1, 60 / 2, 104 / 2] = [21, 30, 52] = 32,760 spacetime tokens.

Look at what that two-stage compression (3D Causal VAE + 1x2x2 Spacetime Bricks) just achieved for our 81-frame video:

• Raw Pixel Attention Pairs (32,348,160^2): 1,046,403,455,385,600 pairs (2,092.8 TB in fp16).
• Spacetime Brick Attention Pairs (32,760^2): 1,073,217,600 pairs (2.15 GB in fp16).
• Total Compute & Memory Reduction: 975,015.2x! That nearly one-million-fold reduction is the single mathematical breakthrough that makes Sora, Wan2.1, and modern video AI possible on real hardware.

How 3D Factorized RoPE Encodes (t’, h, w) Without Losing Direction

When we flatten the 3D token grid [5, 30, 52] into a 1D sequence of 7,800 tokens ([1, 7800, 1536]), the Transformer loses all built-in notion of which tokens are in the same frame, which tokens sit directly above one another, and which tokens sit in the next time slice at the exact same (h, w) screen coordinate.

If we used a standard 1D LLM RoPE (which only counts flat sequence index 0, 1, 2, ..., 7799), moving 1 step forward in time (+1,560 flat positions) would look indistinguishable from wrapping around 30 rows of height!

Instead, WanTransformer3DModel uses 3D Factorized Rotary Position Embedding (3D RoPE). Inside each of the 30 Transformer blocks, the 1,536-dimensional token vector is split across 12 attention heads of 128 dimensions each (12 * 128 = 1,536). Each 128-dimensional query or key vector is partitioned into three dedicated coordinate dials (axes_dim = [44, 42, 42]):

• Dial 1 — Time (t'): First 44 dimensions (22 complex (cos, sin) rotation pairs): Rotated exclusively by the token's temporal slice index t' in {0, 1, 2, 3, 4}.
• Dial 2 — Height (h): Middle 42 dimensions (21 complex rotation pairs): Rotated exclusively by the token's vertical row index h in {0, 1, ..., 29}.
• Dial 3 — Width (w): Final 42 dimensions (21 complex rotation pairs): Rotated exclusively by the token's horizontal column index w in {0, 1, ..., 51}.

Because 2D rotation matrices obey R(a)^T * R(b) = R(b - a), when Query token (t'_1, h_1, w_1) takes a dot product with Key token (t'_2, h_2, w_2), the attention score automatically depends on the exact 3D relative displacement vector (dt' = t'_2 - t'_1, dh = h_2 - h_1, dw = w_2 - w_1)!

Here is a runnable PyTorch snippet implementing Wan2.1’s exact [44, 42, 42] 3D Factorized RoPE and proving that the dot product depends strictly on relative 3D displacement (dt', dh, dw):

import torch

def build_1d_rope_cache(max_pos: int, dim: int, theta: float = 10000.0) -> torch.Tensor:
 """Builds complex rotary frequencies [max_pos, dim // 2] for a single axis."""
 freqs = 1.0 / (theta ** (torch.arange(0, dim, 2, dtype=torch.float64) / dim))
 positions = torch.arange(max_pos, dtype=torch.float64)
 angles = torch.outer(positions, freqs) # [max_pos, dim // 2]
 return torch.polar(torch.ones_like(angles), angles).to(torch.complex64)def apply_3d_rope(x: torch.Tensor, t_idx: int, h_idx: int, w_idx: int, caches) -> torch.Tensor:
 """
 Applies 3D Factorized RoPE with axes_dim = [44, 42, 42] (sum = 128 head_dim).
 x shape: [128] float32 query/key vector for a single head.
 """
 t_cache, h_cache, w_cache = caches
 # Split 128-D vector into [44, 42, 42] and view as complex pairs [22, 21, 21]
 x_t = torch.view_as_complex(x[0:44].reshape(22, 2)) * t_cache[t_idx]
 x_h = torch.view_as_complex(x[44:86].reshape(21, 2)) * h_cache[h_idx]
 x_w = torch.view_as_complex(x[86:128].reshape(21, 2)) * w_cache[w_idx]
 out_c = torch.cat([x_t, x_h, x_w], dim=0)
 return torch.view_as_real(out_c).flatten()caches = (build_1d_rope_cache(21, 44), build_1d_rope_cache(60, 42), build_1d_rope_cache(104, 42))
q_base = torch.randn(128)
k_base = torch.randn(128)# Pair A: token at (t'=1, h=10, w=20) attending to (t'=3, h=12, w=25) -> delta = (+2, +2, +5)
q1 = apply_3d_rope(q_base, 1, 10, 20, caches)
k1 = apply_3d_rope(k_base, 3, 12, 25, caches)# Pair B: shifted in spacetime to (t'=2, h=15, w=30) and (t'=4, h=17, w=35) -> same delta = (+2, +2, +5)
q2 = apply_3d_rope(q_base, 2, 15, 30, caches)
k2 = apply_3d_rope(k_base, 4, 17, 35, caches)print("Dot product A:", torch.dot(q1, k1).item())
print("Dot product B (shifted by same 3D delta):", torch.dot(q2, k2).item())
print("Absolute difference:", (torch.dot(q1, k1) - torch.dot(q2, k2)).abs().item())

5. Inside the 30-Block 3D Diffusion Transformer (WanTransformer3DModel)

Direct Answer: Each of the 30 blocks in WanTransformer3DModel (1.419 billion parameters total, or 1,418,996,800 exact parameters) processes all 7,800 spacetime tokens through three core operations: Full 3D Spatiotemporal Self-Attention (where every brick in every frame attends to all 7,800 bricks across the entire video), UMT5 Text Cross-Attention (where spacetime bricks attend to 512 text prompt embeddings from a 5.681B-parameter UMT5EncoderModel), and 6-Way AdaLN-Zero Timestep Modulation (which scales and shifts activations using the current flow noise level sigma_t).

Let’s trace the exact parameter budget and tensor shapes inside a single block of WanTransformer3DModel (1,418,996,800 parameters across 30 layers):

• Input Spacetime Sequence: [1, 7800, 1536] (7,800 bricks for a 17-frame clip, each with 1,536 channels).
• Sub-Layer 1 — Full 3D Self-Attention (attn1): Projects the 7,800 tokens into 12 heads of 128-D Queries, Keys, and Values ([1, 12, 7800, 128]), applies RMSNorm to Q and K, rotates Q and K with 3D Factorized RoPE ([44, 42, 42]), and computes a [7800, 7800] attention matrix (60,840,000 spacetime links per head). Here, a brick on the car's rear wheel at t' = 0 directly exchanges information with a brick at t' = 4.
• Sub-Layer 2 — Text Prompt Cross-Attention (attn2): Before denoising begins, the user's text prompt is encoded once by UMT5-XXL (UMT5EncoderModel, 5.681 billion parameters, 4,096-D hidden state projected to 1,536-D). In Cross-Attention, the 7,800 video bricks supply the Queries, while the text tokens supply the Keys and Values, steering which objects, sunset lighting, and camera movements materialize at each (t', h, w) coordinate.
• Sub-Layer 3 — Feed-Forward Network (ffn): Expands each 1,536-D brick to an intermediate dimension of 8,960 (5.83x expansion) with a GELU-approximate activation and projects back to 1,536-D.
• 6-Way AdaLN-Zero Timestep Modulation (scale_shift_table): Because the network needs to behave very differently at pure noise (sigma = 1.0, carving global motion trajectories) than at the final step (sigma = 0.0963, sharpening ocean spray and cliff textures), each block learns a [1, 6, 1536] parameter table added to the sinusoidal timestep projection to produce 6 modulation vectors (shift_msa, scale_msa, gate_msa, shift_mlp, scale_mlp, gate_mlp).

6. Simultaneous 4D Rectified Flow Matching and the Flow Shift (s = 3.0) Schedule

Direct Answer: A Video DiT does not generate Frame 1, then Frame 2, then Frame 3. Instead, it initializes the entire [1, 16, 5, 60, 104] 4D volume as pure Gaussian noise (sigma = 1.0) and integrates a straight-line Rectified Flow velocity field v_theta(z_t, sigma_t, c) across 30 Euler steps simultaneously for all frames. Because locking consistent global motion across 7,800 or 32,760 tokens requires spending more steps at high noise levels, Wan2.1 applies a Flow Shift transformation (s = 3.0) (sigma_s = (s * sigma) / (1 + (s - 1) * sigma)), dedicating 53.3% of all steps (16 of 30) to the high-noise regime (sigma > 0.75) compared to just 23.3% (7 of 30) for an unshifted schedule.

If you are used to Large Language Models like GPT-4 or Claude, you are used to autoregressive generation: predicting token t after token t - 1 is completely finished.

Modern Video Diffusion Transformers work in a fundamentally different way: Simultaneous 4D Rectified Flow Matching.

Let z_0 be the clean 4D video latent ([1, 16, 5, 60, 104]) and epsilon be a 4D tensor of standard normal Gaussian noise (N(0, I)) of the exact same shape. Rectified Flow defines a straight-line interpolation path between clean video z_0 (sigma = 0) and pure noise epsilon (sigma = 1):

z_t = (1 - sigma_t) * z_0 + sigma_t * epsilon

Taking the derivative with respect to sigma_t, the constant target velocity vector pointing from clean data to noise along that straight line is simply:

v_target = epsilon - z_0

During inference, we start at sigma_1 = 1.0000 (z_1 = epsilon, 100% static across all 5 temporal slices) and take 30 discrete Euler steps down to sigma = 0.0:

z_{t_next} = z_t + (sigma_next - sigma_t) * v_theta(z_t, sigma_t, prompt)

We hooked into the 30-step Euler loop of Wan2.1-T2V-1.3B on our GPU (M4 in measure_video_gen.py), projected the intermediate latent estimate z_0_hat = z_t - sigma_t * v_theta through the 3D Causal VAE decoder at 6 checkpoints, and measured both PSNR to the final video and Temporal Motion Lock (the Pearson correlation of the decoded start-to-end RGB displacement field Frame 16 - Frame 0, downsampled to the 52x30 patch grid, against the final Step 30 motion displacement field):

• Step 1 (sigma = 1.0000): PSNR to final video = 11.50 dB; Temporal Motion Lock = -0.0016 (0% motion structure; pure random static across all frames).
• Step 6 (sigma = 0.9376): PSNR to final video = 18.02 dB; Temporal Motion Lock = 0.3117 (31.2%). Even though sigma has only dropped by 6.2%, the coastal cliff highway, the red silhouette of the car, and the forward driving direction have already emerged simultaneously across all 17 frames!
• Step 10 (sigma = 0.8752): PSNR to final video = 21.22 dB; Temporal Motion Lock = 0.4704 (47.0%).
• Step 15 (sigma = 0.7745): PSNR to final video = 24.23 dB; Temporal Motion Lock = 0.7389 (73.9%). Nearly three-quarters of the entire video's temporal motion trajectory is locked in halfway through the loop.
• Step 22 (sigma = 0.5633): PSNR to final video = 28.09 dB; Temporal Motion Lock = 0.8965 (89.7%).
• Step 30 (sigma = 0.0963): PSNR to final video = 72.13 dB; Temporal Motion Lock = 1.0000 (100.0%).

Why Video Models Shift the Noise Schedule (flow_shift s = 3.0)

Look closely at the noise level at Step 15 out of 30 above: sigma is 0.7745, not 0.5000! Why is the noise level still above 77% halfway through the 30-step loop?

As shown by Esser et al. (Stable Diffusion 3) and Polyak et al. (Movie Gen), when you increase the number of tokens in a diffusion volume from a single image (1,560 tokens) to a 4D video (7,800 to 32,760 tokens), a constant noise level sigma leaves far more redundant low-frequency signal intact across neighboring frames. If you use a naive linear schedule (s = 1.0), the model rushes past the critical high-noise window (sigma in [0.75, 1.0]) in just a few steps — before the 30 layers of self-attention have time to synchronize where the car is moving across all 81 frames!

To fix this, UniPCMultistepScheduler in Wan2.1 warps the linear timestep schedule sigma using the Flow Shift transformation with shift factor s = 3.0:

sigma_s = (s * sigma) / (1 + (s - 1) * sigma)

On our 30-step GPU schedule:

• Unshifted Linear Schedule (s = 1.0): Only 7 out of 30 steps (23.3%) execute in the high-noise motion-planning band (sigma > 0.75).
• Wan2.1 Shifted Schedule (s = 3.0): 16 out of 30 steps (53.3%) execute in the high-noise band (sigma > 0.75), giving the 3D DiT more than 2.28x as many passes to lock in global camera motion and object trajectories before spending the final 14 steps sharpening textures.

7. Probing the [7800 x 7800] Matrix: How 90.4% of Attention Reaches Across Time

Direct Answer: By registering a forward hook across WanTransformer3DModel (layers 0, 7, 14, 21, 29) at Step 15 (sigma = 0.7745) and probing Query token 4082 (t' = 2, h = 18, w = 26, sitting directly on the moving car), we caught a specialized Motion-Tracking Attention Head (Layer 21, Head 7) in the act: it allocates only 9.63% of its attention weight to tokens in its own time slice (t' = 2) and routes 90.37% of its attention across past and future slices (16.83% to t' = 0, 22.17% to t' = 1, 40.44% to t' = 3, and 10.93% to t' = 4), locking onto the displaced coordinate (h = 17, w = 26) in the next time slice.

How does a Transformer actually keep the red sports car looking like the same car from Frame 0 to Frame 16 while it drives along the highway?

To find out without guessing, we attached a PyTorch hook to compute the exact [12, 7800] softmax attention row for Query Token #4082 — located in the middle temporal slice t' = 2 (Frames 5..8), row h = 18, column w = 26, right on the rear wheel and fender of the moving red sports car — at denoising Step 15 (sigma = 0.7745) across five evenly spaced transformer blocks (layers 0, 7, 14, 21, 29 in M5 of measure_video_gen.py).

For each attention head, we summed the softmax probabilities across the 1,560 tokens in each of the 5 temporal slices (t' = 0, 1, 2, 3, 4). What we found reveals a striking functional specialization inside the 30-layer network:

• Layer 0, Head 0 (Early Broad-Context Head): Distributes attention almost completely uniformly across all 5 time slices: 20.29% in t' = 0, 20.15% in t' = 1, 19.83% in t' = 2 (self-slice), 20.01% in t' = 3, and 19.72% in t' = 4. In Layer 0, the model is simply averaging global scene statistics (overall sunset lighting, cliff color, horizon height).
• Layer 14, Head 4 & Layer 29, Head 5 (Spatial Intra-Frame Heads): Focus 27.11% (Layer 14, Head 4) and 29.11% (Layer 29, Head 5) of their attention inside their own time slice t' = 2 to refine local car body geometry.
• Layer 21, Head 7 (Dedicated Cross-Time Motion-Tracking Head): Allocates only 9.63% (0.0963) of its attention to its own time slice t' = 2 and fires 90.37% (0.9037) of its attention across past and future time slices! Specifically, its temporal distribution and peak target coordinates across the 5 slices are:
• Slice t' = 0 (Frame 0): 16.83% (0.1683), peaking at spatial coordinate (h = 18, w = 26).
• Slice t' = 1 (Frames 1..4): 22.17% (0.2217), peaking at (h = 17, w = 26).
• Slice t' = 2 (Frames 5..8, Query's own slice): 9.63% (0.0963), peaking at (h = 16, w = 21).
• Slice t' = 3 (Frames 9..12, immediate future slice): 40.44% (0.4044), peaking at (h = 17, w = 26) — shifted up by -1 row (-16 RGB pixels) from h = 18 along the car's exact forward highway trajectory!
• Slice t' = 4 (Frames 13..16): 10.93% (0.1093), peaking at (h = 16, w = 21).

Without a single line of hardcoded optical-flow equations, Layer 21 Head 7 learned to act as a 3D trajectory matcher: it finds where the car is located in t' = 1 and t' = 3, copies the wheel and fender features along that diagonal path in spacetime, and forces all 5 slices to agree!

Here is the runnable PyTorch code to extract and partition a [7800] query attention distribution across the 5 temporal slices of a 3D Video DiT:

import math
import torch

def probe_spatiotemporal_attention(
 q_all: torch.Tensor, k_all: torch.Tensor, query_coord=(2, 18, 26), grid_shape=(5, 30, 52)
):
 """
 Computes the exact [num_heads, 7800] softmax attention row for a single
 spacetime brick (t', h, w) and aggregates mass across the T'=5 time slices.
 q_all, k_all shape: [1, num_heads, 7800, head_dim] (after RMSNorm + 3D RoPE)
 """
 T_p, H_p, W_p = grid_shape
 tokens_per_slice = H_p * W_p # 30 * 52 = 1,560 tokens per slice
 t_q, h_q, w_q = query_coord
 flat_idx = t_q * tokens_per_slice + h_q * W_p + w_q # 2*1560 + 18*52 + 26 = 4082 # Compute dot-product attention for ONLY query token #4082 against all 7,800 keys
 q_single = q_all[:, :, flat_idx : flat_idx + 1, :] # [1, H, 1, D]
 scale = 1.0 / math.sqrt(q_all.shape[-1])
 logits = torch.matmul(q_single.float(), k_all.float().transpose(-2, -1)) * scale
 probs = torch.softmax(logits, dim=-1)[0, :, 0, :] # [num_heads, 7800] # Reshape into [num_heads, T'=5, H'=30, W'=52] and sum over (H', W') per time slice
 probs_3d = probs.view(-1, T_p, H_p, W_p)
 slice_mass = probs_3d.sum(dim=(2, 3)) # [num_heads, 5]
 cross_time_ratio = 1.0 - slice_mass[:, t_q] # Fraction of attention outside t'=2
 return flat_idx, slice_mass, cross_time_ratio# Demo with synthetic 12-head, 7800-token Q/K tensors
q_demo = torch.randn(1, 12, 7800, 128)
k_demo = torch.randn(1, 12, 7800, 128)
idx, mass, cross_ratio = probe_spatiotemporal_attention(q_demo, k_demo)
print(f"Probed Query Brick #{idx} -> Slice mass shape: {tuple(mass.shape)}")

8. Classifier-Free Guidance (CFG) in 4D: Steering Velocity Without Oversaturating Time

Direct Answer: In video generation, Classifier-Free Guidance (v = v_uncond + w * (v_cond - v_uncond)) steers not only spatial appearance (RGB contrast) but also temporal motion speed (optical flow magnitude). Sweeping guidance scale w in {1.0, 3.0, 5.0, 9.0} on our GPU shows that moving from w = 1.0 to w = 3.0..5.0 accelerates car motion by 2.46x (0.811 px to 1.999 px mean optical flow), whereas pushing to w = 9.0 overshoots the data manifold — spiking clipped/saturated pixels by 34x over w = 3.0 (0.20% to 6.83%, or 62x over w = 1.0) and cooking the video with neon burn-in artifacts.

At every Euler step, why does Wan2.1-T2V-1.3B actually run two forward passes through its 30 transformer layers (60 passes total over 30 steps)?

Just like in image diffusion, a conditional pass v_cond = v_theta(z_t, sigma_t, prompt) alone often produces washed-out colors and sluggish motion because the velocity field averages over many plausible continuations. To sharpen adherence to the prompt, we also run an unconditional pass v_uncond = v_theta(z_t, sigma_t, negative_prompt) and extrapolate along the difference vector with guidance scale w:

v_cfg = v_uncond + w * (v_cond - v_uncond)

Because v_cond and v_uncond are 4D tensors ([1, 16, 5, 60, 104]), the guidance vector v_cond - v_uncond points simultaneously along spatial detail axes and temporal motion axes! We swept w in {1.0, 3.0, 5.0, 9.0} from the same initial noise seed (M6 in measure_video_gen.py, using 20 Euler steps for w = 1, 3, 9 and our 30-step baseline for w = 5) and measured three physical quantities on the decoded 17-frame videos:

• w = 1.0 (No CFG extrapolation): Mean optical flow magnitude = 0.811 px/frame (sluggish drift), RGB contrast standard deviation = 49.15, and extreme/saturated pixels (<= 2 or >= 253 on a 0..255 scale) = 0.11%.
• w = 3.0 (Balanced Sweet Spot): Mean optical flow magnitude jumps 2.46x to 1.999 px/frame, RGB contrast rises to 67.10, and saturated pixels remain tiny at 0.20%.
• w = 5.0 (Default High-Detail Sweet Spot): Mean optical flow magnitude = 1.775 px/frame, RGB contrast = 69.67, and saturated pixels = 2.19%.
• w = 9.0 (Over-Guided Manifold Overshoot): Mean optical flow spikes to 2.893 px/frame (jittery, hyperactive motion), RGB contrast hits 80.54, and saturated/clipped pixels explode 34x higher than w = 3.0 (and 62x higher than w = 1.0) to 6.83% — burning clipped orange and magenta patches into the sunset sky and cliff highway reflections.

Here is the complete, runnable PyTorch implementation of the 4D Rectified Flow Matching denoising loop with Flow Shift (s = 3.0) and Classifier-Free Guidance (w = 5.0):

import torch

def get_shifted_sigmas(num_steps: int = 30, shift: float = 3.0) -> torch.Tensor:
 """Computes the Wan2.1 UniPC flow-shifted noise schedule sigma_s across 30 steps."""
 sigmas = torch.linspace(1.0, 1.0 / 1000.0, num_steps + 1)[:-1]
 sigmas = (shift * sigmas) / (1.0 + (shift - 1.0) * sigmas)
 return sigmas@torch.no_grad()
def rectified_flow_4d_step(
 z_t: torch.Tensor,
 sigma_t: float,
 sigma_next: float,
 v_cond: torch.Tensor,
 v_uncond: torch.Tensor,
 cfg_scale: float = 5.0,
) -> tuple[torch.Tensor, torch.Tensor]:
 """
 Executes one 4D Rectified Flow Euler update across [B, 16, T', H', W'] simultaneously.
 Returns (z_next, z0_hat_clean_estimate).
 """
 # 1. Extrapolate 4D spatiotemporal velocity vector via Classifier-Free Guidance
 v_cfg = v_uncond + cfg_scale * (v_cond - v_uncond)
 # 2. Estimate clean 4D latent volume z_0_hat = z_t - sigma_t * v_cfg
 z0_hat = z_t - sigma_t * v_cfg
 # 3. Take Euler step along straight-line trajectory toward sigma_next
 dt = sigma_next - sigma_t # Negative step size as sigma decreases from 1 -> 0
 z_next = z_t + dt * v_cfg
 return z_next, z0_hatsigmas = get_shifted_sigmas(num_steps=30, shift=3.0)
print(f"Step 1 sigma: {sigmas[0]:.4f} | Step 15 sigma: {sigmas[14]:.4f} | Step 30 sigma: {sigmas[29]:.4f}")
print(f"Steps with sigma > 0.75: {int((sigmas > 0.75).sum().item())} / 30")

9. How Image-to-Video (I2V) Works: Anchoring Slice t’ = 0

Direct Answer: Because AutoencoderKLWan isolates Frame 0 inside latent slice t' = 0 with zero causal mixing from frames 1..16, we can turn a Text-to-Video model into an Image-to-Video (I2V) generator by encoding a user's reference photograph into z_ref and pinning z_t[:, :, 0:1, :, :] to its exact flow-matched reference trajectory (1 - sigma_next) * z_ref + sigma_next * epsilon_0 after every step. On our GPU, anchoring t' = 0 locks Frame 0 at 44.52 dB PSNR (a +33.77 dB gain over unanchored generation) and pulls subsequent slices t' = 1..4 toward the reference image via 3D self-attention (+1.97 dB PSNR gain across Frames 4..16), while dedicated I2V models go one step further by concatenating [z_t, mask, z_cond] (36 channels) across all time slices.

Now we can see why the asymmetric 1 + (T - 1) / 4 design of the 3D Causal VAE in Section 3 is so clever.

Suppose a user uploads a single photograph ([1, 3, 1, 480, 832]) and asks the AI to animate the scene for 5 seconds. Because the 3D Causal VAE encoder processes Frame 0 independently into latent slice t' = 0 ([1, 16, 1, 60, 104]), there are two ways to condition the 3D DiT on that starting photograph:

1. Inference-Time Slice t' = 0 Trajectory Pinning (Zero-Shot on a 16-Channel T2V Model): Before the Euler loop begins, we encode the reference photograph into z_ref ([1, 16, 1, 60, 104]). At each denoising step (M7 in measure_video_gen.py, 20 steps), after updating z_t, we pin slice t' = 0 along the exact rectified flow line for that noise level: z_t[:, :, 0:1] = (1 - sigma_next) * z_ref + sigma_next * epsilon_0 (reaching clean z_ref at the final step). On our GPU, Frame 0 locked onto the reference photograph at 44.52 dB PSNR (compared to 10.75 dB unanchored — a +33.77 dB jump). Furthermore, because tokens in slices t' = 1..4 attend to slice t' = 0 inside every 3D self-attention layer, the reference colors and layout propagated forward across the clip, lifting PSNR on Frames 4, 8, 12, 16 by +1.97 dB (13.06–13.28 dB vs 10.96–11.29 dB).
2. Dedicated 36-Channel I2V Concatenation (Wan2.1-I2V-14B): Notice why pinning only t' = 0 inside a 16-channel T2V model still drops from 44.52 dB on Frame 0 to 13.28 dB on later frames: slices t' = 1..4 only see the reference image indirectly through noisy self-attention. Dedicated I2V models eliminate this drop by expanding the patch embedding input from 16 channels to 36 channels (16 noisy latent channels z_t + 4 binary mask channels indicating which frames are conditioned + 16 clean reference latent channels z_cond zero-padded across t' = 1..20).

10. Why Video AI Still Breaks Physics: Missing Depth (Z) and the 71.4% Occlusion Wipeout

Direct Answer: Despite generating photorealistic reflections and smooth camera motion, current Video DiTs regularly fail basic object permanence — causing objects that should pass behind a pillar to stall, vanish, or distort. We proved why on our GPU with two experiments (M8): first, 3D RoPE only encodes 2D screen coordinates plus time ([t', h, w]) with zero 3D camera depth axis Z; second, when a foreground stone pillar occludes an object in latent slices t' = 2 and t' = 3, the pillar's pixels physically overwrite the object's 1,536-D spacetime tube vectors at (h, w) — collapsing feature cosine similarity to t' = 1 from 0.9760 down to 0.2792 (a 71.4% signal wipeout) and severing the cross-time attention bridge needed to re-materialize the object at t' = 4.

If 3D Causal VAEs, Spacetime Tubes, 3D RoPE, and 90.4% cross-time self-attention are so powerful, why do Sora, Wan2.1, Kling, and Veo still make bizarre physics blunders — like a person walking behind a signpost and disappearing, or a rolling ball refusing to pass behind a pillar?

To catch this failure mode in the wild, we prompted Wan2.1-T2V-1.3B on our GPU with a classic physical occlusion test (M8 in measure_video_gen.py):

“A bright yellow tennis ball rolling left to right across a wooden table, passing behind a dark vertical stone pillar in the center of the frame and emerging on the right side, static camera, photorealistic.”

What did the 1.419-billion-parameter 3D DiT actually generate across the 17 frames (shown in the top strip of Figure 12a)?

• Frames 0 to 12 (13 consecutive frames): Instead of rolling the ball behind the stone pillar from left to right, the model spawns the yellow tennis ball directly in front of the center stone pillar base and stalls it in place, slightly swelling its radius because the network refuses to let the ball’s pixels disappear behind the stone texture!
• Frame 16: On the final slice, instead of emerging cleanly on the right side of the pillar, the ball suddenly lunges toward the camera in a streak of yellow motion blur while the background table warps.

Why can’t a 3D Diffusion Transformer track a rolling ball behind a stone pillar the way a 1990s video game engine can? Look back at the mathematics we just derived in Sections 3, 4, and 7:

• 1. Missing Coordinate Axis — No Depth (Z): In a 3D game engine (like Unreal Engine or Blender), the tennis ball is a 3D mesh at depth Z = 2.5 meters and the stone pillar is a separate 3D mesh at Z = 1.2 meters. When the ball rolls behind the pillar, the ball still exists at (X, Y, Z = 2.5) in memory — the renderer simply draws the closer Z = 1.2 pixels on top. Look at our 3D RoPE dials in Section 4: [Time t' (44 dims), Height h (42 dims), Width w (42 dims)]. There is no Depth dial Z! The model only has a single flat 2D grid (h, w) per time slice t'. Two objects at the same screen location (h, w) cannot occupy separate depth layers in the latent tensor.
• 2. The 71.4% Latent Matrix Overwrite: To measure exactly what happens inside the spacetime tokens when an occluder covers an object, we ran a controlled occlusion experiment on our GPU (bottom strip of Figure 12a and Figure 12b): we placed a dark vertical stone pillar mask (x = 320..512, y = 120..420) over the center of our car video during Frames 5 to 11 (which map to latent slices t' = 2 and t' = 3), passed both the unoccluded and occluded videos through AutoencoderKLWan and the 1,536-D patch_embedding layer ([1, 5, 30, 52, 1536]), and measured the cosine similarity of the center tube vectors relative to pre-occlusion slice t' = 1:
• Unoccluded Video Cosine Similarity to t' = 1 across t' = [0, 1, 2, 3, 4]: [0.9244, 1.0000, 0.9760, 0.9754, 0.9601]. Every interior time slice t' = 1..4 preserves > 96.0% feature similarity, giving Layer 21 Head 7 a continuous similarity bridge across time.
• Occluded Video Cosine Similarity to t' = 1 across t' = [0, 1, 2, 3, 4]: [0.9244, 1.0000, 0.2792, 0.4927, 0.9240]. In slice t' = 2, the stone pillar's dark gray texture overwrites the underlying object's 1,536-D spacetime tube vector, crashing cosine similarity from 0.9760 down to 0.2792 — a 71.4% signal wipeout (and 0.4927, a 49.5% wipeout, in t' = 3)!
• 3. Broken Cross-Time Attention Bridge: Remember from Section 7 that motion-tracking heads like Layer 21 Head 7 link slice t' = 2 to adjacent slice t' = 3 (40.44% attention weight) by matching visual features (Q * K^T) across neighboring slices. When the object disappears behind a pillar for 7 to 12 frames (t' = 2 and t' = 3), those intermediate slices contain only stone pillar features. For an emerging object at t' = 4 to know its exact velocity, spin, and appearance from t' = 1, self-attention has to bridge across a dead zone of overwritten stone tokens without any explicit 1/2 * m * v^2 momentum state or 3D scene graph!

11. Complete End-to-End Engineering Blueprint of a Modern Video DiT

Direct Answer: Putting all 10 stages together, generating a 17-frame (832x480) or 81-frame widescreen video on a single GPU follows a deterministic 6-step pipeline: UMT5-XXL encodes the text prompt ([1, 512, 4096]), a 4D Gaussian latent tensor ([1, 16, T', 60, 104]) is initialized at sigma = 1.0, patchified into 1x2x2 spacetime bricks (1,536-D) with [44, 42, 42] 3D RoPE, denoised across 30 flow-shifted (s = 3.0) Euler steps through 30 DiT blocks with CFG (w = 5.0), and finally decoded through the 3D Causal VAE (1 + (T - 1) / 4) into coherent RGB frames at 27.74 dB temporal PSNR.

Here is the complete numerical reference summary of every tensor shape, compression ratio, and GPU measurement from our experiments (data/video_gen_results.json):

• Hardware & Runtime: NVIDIA RTX PRO 6000 Blackwell Server Edition (94.97 GB VRAM), generating 17 frames (832x480) across 30 Euler steps in 11.67 seconds (1,077.73 ms for 3D Causal VAE encode + decode).
• Text Encoder (UMT5EncoderModel): 5.681 billion parameters, mapping the text prompt into 4,096-D embeddings projected to 1,536-D for Cross-Attention.
• 3D Causal VAE (AutoencoderKLWan): 126.89 million parameters, 16 latent channels, 8x8 spatial compression, 4x causal temporal compression after solo Frame 0 (T' = 1 + (T - 1) / 4), 0.000000 backward future-to-past leakage, 44.53 dB PSNR on Frame 0, and 40.18 dB mean PSNR on interior frames 1..16.
• Spacetime Tube Patchification (patch_size = (1, 2, 2)): Compresses each [16, 1, 2, 2] latent brick (64 floats, spanning 4 x 16 x 16 = 1,024 interior RGB pixels) into a 1,536-D token. Reduces an 81-frame video (32,348,160 pixels) to 32,760 tokens, cutting self-attention pairs from 1.046 quadrillion (2,093 TB) down to 1.073 billion (2.15 GB) — a 975,015.2x reduction.
• 3D Factorized RoPE (axes_dim = [44, 42, 42]): Splits each 128-D attention head into 22 Time pairs (44-D), 21 Height pairs (42-D), and 21 Width pairs (42-D).
• 3D Diffusion Transformer (WanTransformer3DModel): 1,418,996,800 parameters (1.419B) across 30 layers, 12 heads x 128-D, 8,960-D FFN, and 6-way AdaLN-Zero timestep modulation.
• Simultaneous 4D Flow Matching (flow_shift = 3.0, 30 steps): Concentrates 53.3% (16/30) of denoising steps at high noise (sigma > 0.75), locking in 31.2% of the motion trajectory by Step 6 (sigma = 0.9376) and 73.9% by Step 15 (sigma = 0.7745).
• Cross-Time Attention & Physics Limits: Layer 21 Head 7 routes 90.37% of its attention outside its own time slice (40.44% in t' = 3 alone) to track moving objects (27.74 dB temporal PSNR vs 10.12 dB for independent 2D frames), yet remains vulnerable to total 3D occlusion (-71.4% tube cosine similarity wipeout when a foreground pillar overwrites slices t' = 2, 3).

Frequently Asked Questions (FAQ)

Why do AI video generators like Wan2.1 always generate 17, 33, or 81 frames (4k + 1)?

Because the 3D Causal VAE (AutoencoderKLWan) keeps Frame 0 in its own dedicated first latent slice (t' = 0) and compresses all remaining frames in groups of 4 (T' = 1 + (T - 1) / 4). Choosing T = 4k + 1 (17 = 4(4) + 1 -> 5 slices; 81 = 4(20) + 1 -> 21 slices) ensures every interior latent slice contains an exact 4-frame group while keeping t' = 0 pure for high-PSNR first-frame anchoring (44.53 dB) and seamless Image-to-Video conditioning.

Does Sora or Wan2.1 generate video frame-by-frame like an LLM generates words?

No. Standard Diffusion Transformer video models like Sora and Wan2.1 use Simultaneous 4D Rectified Flow Matching: all 5 (or 21) temporal slices start as random Gaussian noise at Step 1 (sigma = 1.0) and are denoised together in parallel across 30 Euler steps. By Step 6 (sigma = 0.9376), 31.2% of the global motion trajectory across the entire clip is already locked in, and by Step 15 (sigma = 0.7745), 73.9% of the motion trajectory is locked.

Why do objects disappear or morph when they pass behind a pillar in AI video?

Because the model operates on a compressed 3D grid of 2D screen coordinates plus time ([t', h, w]) with no 3D depth coordinate (Z). When a stone pillar covers a moving object at screen position (h, w) in intermediate slices t' = 2 and t' = 3, the pillar's latent vector physically overwrites the object's 1,536-D spacetime tube vector at that position — dropping feature cosine similarity from 0.9760 to 0.2792 (a 71.4% wipeout) and breaking the local cross-time attention bridge.

Methodology, Hardware & AI-Generated Media Disclosure

All tensor shapes, PSNR measurements, optical flow statistics, attention matrices, and occlusion benchmarks reported in this article were measured directly by the author in PyTorch on an NVIDIA RTX PRO 6000 Blackwell Server Edition GPU (94.97 GB VRAM) using the open-weight Alibaba Wan2.1-T2V-1.3B (Wan-AI/Wan2.1-T2V-1.3B-Diffusers) model. In accordance with Towards AI publication guidelines, we explicitly disclose that the benchmark video frames shown in Figures 1 through 13 (including the coastal highway red sports car clip and the yellow tennis ball occlusion test) were generated on our GPU by Wan2.1-T2V-1.3B as experimental test outputs for architectural analysis, and all explanatory diagrams and telemetry plots were created by the author using Manim and Python.

References & Primary Literature

1. Wan: Open and Advanced Large-Scale Video Generative Models — Wan Team, Alibaba Group (2025), arXiv:2503.20314.
2. Scalable Diffusion Models with Transformers (DiT) — William Peebles and Saining Xie (2023), ICCV 2023 / arXiv:2212.09748.
3. Video Generation Models as World Simulators (Sora Technical Report) — Tim Brooks, Bill Peebles, Connor Holmes, Deva Ramanan, Aditya Ramesh et al., OpenAI (2024).
4. Flow Matching for Generative Modeling — Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le (2023), ICLR 2023 / arXiv:2210.02747.
5. Scaling Rectified Flow Transformers for High-Resolution Image Synthesis (SD3 & Flow Shift) — Patrick Esser, Sumith Kulal, Andreas Blattmann et al. (2024), ICML 2024 / arXiv:2403.03206.
6. RoFormer: Enhanced Transformer with Rotary Position Embedding (RoPE) — Jianlin Su et al. (2021), arXiv:2104.09864.
7. Movie Gen: A Cast of Media Foundation Models — Adam Polyak et al., Meta AI (2024), arXiv:2410.13720.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI