Why a queue?SimCLR’s negatives are limited to the batch. A queue of past keys holds far more.
Pseudocode
for x in loader: x_q, x_k = aug(x), aug(x)# f_k initialized from f_q q = f_q(x_q) # N x C k = f_k(x_k).detach()# queue: C x K keys, past batches l_pos = (q * k).sum(1, True) # N x 1 l_neg = q @ queue # N x K logits = cat([l_pos, l_neg], 1) labels = zeros(N) # positive at 0 loss = CE(logits / t, labels) loss.backward(); opt.step() # f_q f_k = m * f_k + (1 - m) * f_qenqueue(queue, k); dequeue(queue)# v2: MLP head, blur, cosine lr, t=0.2
Query and key views
crop, color jitter, flip, grayscale (v1)
Two random views of one patch; training pulls them together.
Two encoders
query encoder, trained by backpropagation
key encoder: no gradients, initialized from f_q
The key encoder is a slow, gradient-free copy of the query encoder.
Normalized embeddings
128ResNet-50, then fc to 128-d, ℓ2-normalized
detached; key batch shuffled across GPUs (shuffling BN)
Each view becomes a unit vector: or .
Dictionary as a queue
negatives come from the queue, decoupled from the batch size (256)
The dictionary of negatives can be far bigger than the batch.
InfoNCE over keys
the key of the same image, at index 0
0.07 in v1
Train to find its own key among .
Momentum update
momentum coefficient, 0.999
Slow drift keeps old and new keys in the queue comparable.
Update the queue
queuekeys from recent batches by a slowly changing encoder
This batch’s keys join; the oldest batch leaves.
MoCo v2 changes
Component
MoCo v1
MoCo v2
Projection head
Single linear layer
2-layer MLP with ReLU
Data augmentation
Standard augmentation
Stronger augmentation, including Gaussian blur
Learning-rate schedule
Step decay
Cosine decay
Temperature
0.07
0.2
Barlow Twins
H&E patch (unlabeled)
Two random augmentations(as in BYOL, incl. solarization)
View
random crop, then augment
View
random crop, then augment
Twin AEncoder ResNet-50Projector, 3 layers
Twin B, same weightsEncoder ResNet-50Projector, 3 layers
Shared weights
8,192-d embedding
8,192-d embedding
Cross-correlation of the two embeddingscomputed over the batch; 8 of the 8,192 dimensions shown
batch × , each column standardized
,
Diagonal → 1: each dimension is invariant to the augmentations
Off-diagonal → 0: dimensions carry non-redundant information
Training drives toward the identity
No negatives, predictor, stop-gradient or momentum encoder
Downstream: keep and its 2,048-d output ; drop
Pseudocode
# x: N unlabeled H&E patchesfor x in loader: y_a, y_b = augment(x), augment(x) h_a, h_b = f(y_a), f(y_b) # shared z_a, z_b = g(h_a), g(h_b) # N x D z_a = (z_a - z_a.mean(0)) / z_a.std(0) z_b = (z_b - z_b.mean(0)) / z_b.std(0) c = z_a.T @ z_b / N # D x D on = ((diag(c) - 1) ** 2).sum() off = (off_diagonal(c) ** 2).sum() loss = on + lambd * off loss.backward(); opt.step()# after training: keep f, use h
Two augmented views
crop, flip, color jitter, grayscale, blur, solarization
independent draws; blur and solarization odds differ per view
One encoder, shared weights
encoder (ResNet-50), the same weights for both views
2,048-d representation kept for downstream tasks
A wide projector
projector: 3 linear layers of 8,192 units
BN, ReLUafter the first two layers
Standardize each dimension over the batch
mean and standard deviation of dimension over the batch
a sample in the batch (2,048 in the paper)
Cross-correlation between the views
, entries between −1 and 1
embedding dimensions, not samples
Barlow Twins loss
5 × 10⁻³, weights the redundancy term
invariance: each dimension agrees across views
redundancy reduction: dimensions decorrelate
Target
identity matrix
nonegatives, predictor, stop-gradient or momentum encoder
VICReg
H&E patch (unlabeled)
Two random augmentations(as in BYOL, incl. solarization)
View
random crop, then augment
View
random crop, then augment
BranchEncoder ResNet-50Expander, 3 layers
Branch′Encoder ResNet-50Expander, 3 layers
Shared weights(or not: may differ)
8,192-d, not normalized
8,192-d, not normalized
Invariance
Pull the two embeddings together (MSE)
VarianceKeep each dimension’s std above , per branchCovariance
Decorrelate dimensions within each branch
Component
What it does
Why
Invariance
Makes positive pairs similar
Learn augmentation-invariant representations
Variance
Ensures dimensions vary across images
Prevent collapse
Covariance
Reduces correlation between dimensions
Reduce redundancy
No negatives, stop-gradient, predictor or embedding normalization
Downstream: keep and its 2,048-d output ; drop
Pseudocode
for x in loader: x_a, x_b = augment(x), augment(x) y_a, y_b = f(x_a), f(x_b) z_a, z_b = h(y_a), h(y_b) # N x D sim = mse_loss(z_a, z_b); v = c = 0for z in (z_a, z_b): # each branch std = sqrt(z.var(0) + 1e-4) v += relu(1 - std).mean()for z in (z_a, z_b): C = cov(z) # D x D c += off_diag(C).pow(2).sum() / D loss = 25 * sim + 25 * v + 1 * c loss.backward(); opt.step()
Two augmented views
the image; and are its two views
crop, flip, color jitter, grayscale, blur, solarization
An encoder per branch
encoder (ResNet-50); its 2,048-d output is kept
usually equal to , but the branches may differ
Expander
expander: 3 layers of 8,192 units, discarded after training
a batch of embeddings, not normalized
Invariance
batch size (2,048 in the paper)
the only term that links the two branches
Invariance: Pulls two augmented views of the same image closer together.
Variance (hinge on the std)
target standard deviation, 1
10⁻⁴, keeps the square root stable
dimension across the batch, per branch
Variance: Prevents different images from all being mapped to the same representation.
Covariance, within each branch
covariance within one branch
vs. BarlowBarlow Twins correlates the two branches
Covariance: Encourages different embedding dimensions to carry nonredundant information.
Total loss
ImageNet defaults
nonegatives, stop-gradient, predictor or normalization
Invariance: Pulls two augmented views of the same image closer together.
Variance: Prevents different images from all being mapped to the same representation.
Covariance: Encourages different embedding dimensions to carry nonredundant information.
DINO
H&E patch (unlabeled)
Multi-crop augmentation(crop, color jitter, blur, …)
Student views (global + local crops)
2 global crops (high resolution)
•••
Several local crops (low resolution)
Global crops only(the student’s 2 global crops)
Teacher views (global only)
2 global crops (high resolution)
Student NetworkBackboneVision TransformerProjection head
Teacher NetworkBackboneVision TransformerProjection head
Cross-entropy between teacher and student output distributions from different views
Cross-entropy
EMA update
Pseudocode
# x: unlabeled H&E patchfor x in wsi_patches: crops = multi_crop(x) v_s = crops # 2 global + n local v_t = crops[:2] # same globals s = student(v_s)# teacher: stop gradientwith no_grad(): t = teacher(v_t) p_s = softmax(s / tau_s)# center, then sharpen p_t = softmax((t - c) / tau_t)# teacher global i vs student view j loss = mean(H(p_t[i], p_s[j])for i in (0, 1)for j in views if j != i) loss.backward() optimizer.step() # student teacher = ema(teacher, student) c = ema(c, t.mean(0))
Multi-crop views
all views of patch
2 global crops, 224 px
local crops, 96 px
number of local crops
Teacher views
the same 2 global crops
all views the student sees
Stop gradient
teacher weights, never trained by the loss
DINO loss
Student softmax
student logit for one view
output dimensions, 65,536 in DINO
student temperature, 0.1
Centered, sharpened teacher softmax
teacher logit
center, a running mean of
teacher temperature, warmed up from 0.04 to 0.07, below τₛ
Cross-entropy across views
cross-entropy
a teacher global crop; any other student view
EMA updates
student and teacher weights
momentum, 0.996 rising to 1
center momentum, 0.9 in the official code
batch mean of teacher outputs
MAE
H&E patch (unlabeled)
196 patches of 16 × 16 px
75% masked at random
25% visible
EncoderViT, visible patches only
Encoded tokens + shared mask token, with positions
Decoderlightweight
Masked patches predicted
Loss on masked patches onlymean squared error, per-patch normalized pixelspredictionvs.target
After pretraining, drop the decoder and run the encoder on all patches
Pseudocode
# x: N unlabeled H&E patchesfor x in loader: p = patchify(x) # B x 196 x 768 keep, mask = random_masking(0.75) z = encoder(p[keep] + pos[keep]) t = cat([z, m.repeat(n_masked)]) t = unshuffle(t) + pos p_hat = decoder(t)# p_hat: pixels for all 196 patches y = (p - p.mean(-1)) / p.std(-1) err = ((p_hat - y) ** 2).mean(-1) loss = err[mask].mean() loss.backward(); opt.step()# after: keep encoder, drop decoder
Patchify
one 16 × 16 px patch (7 × 7 of 196 shown)
Random masking
masked patches, sampled uniformly at random
Encoder on visible patches only
ViT encoder (ViT-B/16, L/16, H/14)
positional embedding of patch
25%of tokens enter the encoder, so pretraining is cheaper
Mask tokens join the sequence
one shared, learned mask token
Lightweight decoder
decoder: 8 blocks, 512-d, about 9% of ViT-L’s FLOPs per token
Reconstructed pixels
768= 16 × 16 × 3 pixel values per patch
Loss on masked patches
mean and std of patch i’s own pixels
the loss is computed on masked patches only
Downstream
discarded after pretraining
aug.random crop and flip only
iBOT
H&E patch (unlabeled)
Multi-crop + block masking(masks on the 2 global crops)
Student views (masked global + local crops)
2 global crops, 224 px (patches masked)
•••
Local crops, 96 px (no masks)
Global crops only(same 2 crops, no masks)
Teacher views (global only)
2 global crops, 224 px (unmasked)
Student NetworkBackboneViT, patch 16CLSpatchesShared head
Teacher NetworkBackboneViT, patch 16CLSpatchesShared head