DINOv2 for pathology intuition

KoLeo vs KDE: why swap the regularizer?

Both push embeddings apart so tiles don’t collapse to one point. They differ when two embeddings are almost identical.

KoLeoKDE kernel
penalty = 0.69 = 4.38
gradient = −2 = −2.5
as explodesstays finite, gradient → 0

von Mises–Fisher (vMF) kernel

What is vMF?
5
0 · uniform directions20 · tight concentration

Dots are illustrative vMF samples on a 3D unit sphere. The arrow marks the preferred direction μ.

View reference image
Points distributed over a unit sphere, with dense blue and green clusters around different directions.
Illustration of directional data on a sphere.

von Mises–Fisher (vMF) naturally models directions on a hypersphere.

zi⊤zj=cos(θij)

For normalized embeddings, the dot product measures angular similarity. It depends on the angle between their directions, with each embedding having unit magnitude.

The von Mises–Fisher distribution describes directions on the unit sphere. A preferred direction sets the center of concentration, and κ controls how tightly points cluster around it.

For normalized embeddings, the vMF kernel gives higher weight to directions with greater cosine similarity. Increasing κ makes this preference sharper.

k(zi,zj)=eκzi⊤zj
zi,zj
Two normalized image embeddings.
zi⊤zj
Cosine similarity.
κ=5
Controls how sharply the kernel responds to similarity.

For unit-normalized embeddings, d2=2−2zi⊤zj. Substituting gives

k(d)=eκe−κd2/2

A smooth Gaussian-shaped distance profile, highest at zero and decreasing over 0≤d≤2.

KDE aggregates the entire batch

LKDE=1N ∑ilog( ∑j≠i eκzi⊤zj)

For each embedding, sum the kernel similarities to every other embedding in the batch. Take the logarithm, then average over the N anchors.

Minimize this loss to discourage excessive concentration of embeddings. It is the negative of the paper’s kernel entropy estimator.

The distance plot shows one pair’s kernel contribution. The batch loss applies a log to the sum of all contributions for each anchor.

Why a diversity term at all?

Invariance: two views of the same tile should match, .

Diversity: different tiles should not all collapse, . DINO’s own tricks help too; this term adds to them.

KoLeo’s problem in pathology

Tissue repeats across patients: two normal liver tiles can look almost the same, so often. The push then explodes and can split tiles that should stay close.

KDE’s fix: a bounded kernel

The penalty never exceeds , and its gradient fades to 0 near duplicates. Same goal, stable gradients.

KoLeo’s push explodes for near-duplicate tiles. KDE’s push stays finite.

Understanding NT-Xent

One anchor i, its positive view j, and the other views in a batch of N images.

ℓi,j=−logexp(sim(zi,zj)/τ)∑k=12N𝟙[k≠i]exp(sim(zi,zk)/τ)

Increase the fraction. It assigns probability to the positive view j among all 2N−1 candidate views. The denominator includes the positive and 2N−2 negatives. The anchor itself is excluded.

Positive score ÷ all candidate scores → probability of selecting view j
Positive-pair probability pℓi,j = −log(p)
Low probabilityHigh probability

p is the positive-pair probability in the fraction above.

Pair loss −log(p)0.431

Why use a projection head?

The MLP separates learning useful features from optimizing the contrastive loss.

Image xEncoder fhMLP gzContrastive loss

Representation space h

The encoder produces a rich representation of morphology, texture, color, and other image features.

These features can support a range of downstream tasks, including tasks that use information affected by the training augmentations.

Contrastive space z

The projection head transforms h into z, where the contrastive loss is applied.

The loss encourages agreement between augmented views. This can suppress augmentation-specific information, such as stain variations.

A buffer between the objectives

Applying the loss to z gives the projector room to specialize in augmentation invariance, while h can preserve additional information useful downstream.

Gradients still flow through the projector to train the encoder.

After pretraining

SimCLR found that representations before the projection head, h, performed better on downstream evaluation. Discard the projector and retain the encoder for downstream use.

DINO intuition

Why centering and sharpening?

With no labels, the student can match the teacher trivially by giving every image the same answer. Each operation blocks one way of collapsing.

Centering: subtract a running mean

is the average teacher output over recent batches. Subtracting it means no single dimension can win for every image.

On its own, centering pushes the teacher toward a flat, uniform output. A flat target tells the student nothing.

Sharpening: a lower teacher temperature

Dividing by a small makes the teacher’s softmax peaked and confident, sharper than the student’s ().

On its own, sharpening lets one dimension win for every image, so every image looks the same.

Together: confident and diverse

Centering keeps the dimensions balanced across the batch. Sharpening keeps each image’s target confident.

The two pull in opposite directions, so the teacher can’t settle into either collapse.

Each patch gets its own clear target, so the student has something real to learn.

Centering stops one dimension from dominating. Sharpening stops the output from going flat. You need both.