Interview presentation

PRISM’s slide encoder

Perceiver

tilesN × 2,560 latents513 × 1,280 (learned) one block Cross-attention Latent Transformer × 6self-attention among latents × 8sharedweights slide embedding memory for BioGPT

How it works

  1. A small, fixed set of learned latents (513) stands in for the whole slide.
  2. Cross-attention: the latents query all tile embeddings and pull in what they need.
  3. Self-attention among the latents (6 layers) lets them exchange information.
  4. Repeat 8 times. One latent is the slide embedding; the other 512 go to the text decoder.

Cross-attention step

Slide encoder of THREADS, MADELEINE and CHIEF

Attention MIL (ABMIL)

softmaxover tiles slide embedding the same tile embeddings, averaged with these weights
  1. 0.30
  2. 0.22
  3. 0.14
  4. 0.10
  5. 0.08
  6. 0.07
  7. 0.05
  8. 0.04

Example weights for eight tiles (illustrative).

How it works

  1. A small network scores every tile embedding (tanh branch gated by a sigmoid branch).
  2. A softmax over the tiles turns the scores into weights that sum to 1.
  3. The slide embedding is the weighted sum of the tile embeddings.
  4. Multi-head: repeat with several attention heads and concatenate (THREADS 2, MADELEINE 4).

Attention weight

Slide embedding

Prov-GigaPath’s slide encoder

LongNet dilated attention

Rows are queries, columns are keys; 32 tokens shown.

How it works

  1. Split the tile sequence into segments of length ; attention stays inside a segment.
  2. Within each segment keep only every r-th tile (dilation), so long segments stay cheap.
  3. Use several pairs at once: short dense segments for neighbours, long sparse ones for far context.
  4. Mix their outputs. Cost grows linearly with the number of tiles instead of quadratically.

Mixing the patterns

TITAN’s slide encoder

2D ALiBi

Bias from the orange patch to every other patch; darker means less penalty.

How it works

  1. No position embeddings are added to the patch features.
  2. Instead, each attention score is reduced in proportion to the distance between the two patches on the slide grid.
  3. Each head has its own slope: steep heads attend locally, gentle heads attend far.
  4. Because only distance matters, a model trained on 16 × 16 regions also runs on whole slides.

Distance-biased attention

DINOv2 for pathology intuition

KoLeo vs KDE: why swap the regularizer?

Both push embeddings apart so tiles don’t collapse to one point. They differ when two embeddings are almost identical.

KoLeoKDE kernel
penalty = 0.69 = 4.38
gradient = −2 = −2.5
as explodesstays finite, gradient → 0

Why a diversity term at all?

Invariance: two views of the same tile should match, .

Diversity: different tiles should not all collapse, . DINO’s own tricks help too; this term adds to them.

KoLeo’s problem in pathology

Tissue repeats across patients: two normal liver tiles can look almost the same, so often. The push then explodes and can split tiles that should stay close.

KDE’s fix: a bounded kernel

The penalty never exceeds , and its gradient fades to 0 near duplicates. Same goal, stable gradients.

KoLeo’s push explodes for near-duplicate tiles. KDE’s push stays finite.

DINO intuition

Why centering and sharpening?

With no labels, the student can match the teacher trivially by giving every image the same answer. Each operation blocks one way of collapsing.

Centering: subtract a running mean

is the average teacher output over recent batches. Subtracting it means no single dimension can win for every image.

On its own, centering pushes the teacher toward a flat, uniform output. A flat target tells the student nothing.

Sharpening: a lower teacher temperature

Dividing by a small makes the teacher’s softmax peaked and confident, sharper than the student’s ().

On its own, sharpening lets one dimension win for every image, so every image looks the same.

Together: confident and diverse

Centering keeps the dimensions balanced across the batch. Sharpening keeps each image’s target confident.

The two pull in opposite directions, so the teacher can’t settle into either collapse.

Each patch gets its own clear target, so the student has something real to learn.

Centering stops one dimension from dominating. Sharpening stops the output from going flat. You need both.