Interview presentation

PRISM’s slide encoder

Perceiver

tilesN × 2,560 latents513 × 1,280 (learned) one block Cross-attention Latent Transformer × 6self-attention among latents × 8sharedweights slide embedding memory for BioGPT

How it works

  1. A small, fixed set of learned latents (513) stands in for the whole slide.
  2. Cross-attention: the latents query all tile embeddings and pull in what they need.
  3. Self-attention among the latents (6 layers) lets them exchange information.
  4. Repeat 8 times. One latent is the slide embedding; the other 512 go to the text decoder.

Cross-attention step

Slide encoder of THREADS, MADELEINE and CHIEF

Attention MIL (ABMIL)

softmaxover tiles slide embedding the same tile embeddings, averaged with these weights
  1. 0.30
  2. 0.22
  3. 0.14
  4. 0.10
  5. 0.08
  6. 0.07
  7. 0.05
  8. 0.04

Example weights for eight tiles (illustrative).

How it works

  1. A small network scores every tile embedding (tanh branch gated by a sigmoid branch).
  2. A softmax over the tiles turns the scores into weights that sum to 1.
  3. The slide embedding is the weighted sum of the tile embeddings.
  4. Multi-head: repeat with several attention heads and concatenate (THREADS 2, MADELEINE 4).

Attention weight

Slide embedding

Prov-GigaPath’s slide encoder

LongNet dilated attention

Rows are queries, columns are keys; 32 tokens shown.

How it works

  1. Split the tile sequence into segments of length ; attention stays inside a segment.
  2. Within each segment keep only every r-th tile (dilation), so long segments stay cheap.
  3. Use several pairs at once: short dense segments for neighbours, long sparse ones for far context.
  4. Mix their outputs. Cost grows linearly with the number of tiles instead of quadratically.

Mixing the patterns

oFM: episode encoder

GatorTron-base-2k architecture

A clinical BERT from the University of Florida and NVIDIA. Its 24 Transformer blocks read the day’s text, and the [CLS] output becomes the episode token.

[CLS]ecog:[NUM]hemo…[NUM]g/dL…[SEP]Token + position embedding2,048 positions; FoNE at each [NUM]× 24 blocks[CLS] state, 1,024-dbecomes the episode tokenOne block (pre-LayerNorm)LayerNormSelf-attention16 heads × 64-d+LayerNormFeed-forward1,024 → 4,096 → 1,024+
24Transformer blocks
1,024hidden size
16attention heads, 64-d each
4,096feed-forward width
2,048tokens of context (base model: 512)
≈345Mparameters

Inside each block

Each of the 16 heads computes . LayerNorm comes before each sublayer, as in Megatron-BERT.

How it was trained, and how oFM uses it

Pretrained with masked-language modeling on over 90B words, mostly de-identified clinical notes from UF Health, plus PubMed and MIMIC-III.

In oFM it is trained further with FoNE, and its final [CLS] state is the episode token .

oFM intuition

FoNE: how a number becomes a vector

A text tokenizer splits 11.2 into pieces and loses its size. FoNE turns the value into smooth waves, so nearby values get nearby vectors.

1Compress the range

How each term works
log
Compresses large numbers so they don’t dominate the encoding.
|v|
Uses the magnitude so the logarithm can also handle originally negative values.
+1
Makes the transformation work at zero, since log(1)=0.
sign(v)
Preserves whether the original value was positive or negative.

2Read with waves at many speeds

Slow waves track coarse size; fast waves pick out fine differences, like rulers with different tick spacings. Six of 32 shown.

3The code

A
B

32 sines, then 32 cosines; orange is +1, blue is −1. Then MLP + LayerNorm to 1,024-d, added at the [NUM] token.

4Nearby values, nearby codes

0.99similarity of A and B

One token per number, and the vector knows its size: 11.2 sits next to 11.5, far from 250.

DINOv2 for pathology intuition

KoLeo vs KDE: why swap the regularizer?

Both push embeddings apart so tiles don’t collapse to one point. They differ when two embeddings are almost identical.

KoLeoKDE kernel
penalty = 0.69 = 4.38
gradient = −2 = −2.5
as explodesstays finite, gradient → 0

von Mises–Fisher (vMF) kernel

What is vMF?
5
0 · uniform directions20 · tight concentration

Dots are illustrative vMF samples on a 3D unit sphere. The arrow marks the preferred direction μ.

View reference image
Points distributed over a unit sphere, with dense blue and green clusters around different directions.
Illustration of directional data on a sphere.

von Mises–Fisher (vMF) naturally models directions on a hypersphere.

zi⊤zj=cos(θij)

For normalized embeddings, the dot product measures angular similarity. It depends on the angle between their directions, with each embedding having unit magnitude.

The von Mises–Fisher distribution describes directions on the unit sphere. A preferred direction sets the center of concentration, and κ controls how tightly points cluster around it.

For normalized embeddings, the vMF kernel gives higher weight to directions with greater cosine similarity. Increasing κ makes this preference sharper.

k(zi,zj)=eκzi⊤zj
zi,zj
Two normalized image embeddings.
zi⊤zj
Cosine similarity.
κ=5
Controls how sharply the kernel responds to similarity.

For unit-normalized embeddings, d2=2−2zi⊤zj. Substituting gives

k(d)=eκe−κd2/2

A smooth Gaussian-shaped distance profile, highest at zero and decreasing over 0≤d≤2.

KDE aggregates the entire batch

LKDE=1N ∑ilog( ∑j≠i eκzi⊤zj)

For each embedding, sum the kernel similarities to every other embedding in the batch. Take the logarithm, then average over the N anchors.

Minimize this loss to discourage excessive concentration of embeddings. It is the negative of the paper’s kernel entropy estimator.

The distance plot shows one pair’s kernel contribution. The batch loss applies a log to the sum of all contributions for each anchor.

Why a diversity term at all?

Invariance: two views of the same tile should match, .

Diversity: different tiles should not all collapse, . DINO’s own tricks help too; this term adds to them.

KoLeo’s problem in pathology

Tissue repeats across patients: two normal liver tiles can look almost the same, so often. The push then explodes and can split tiles that should stay close.

KDE’s fix: a bounded kernel

The penalty never exceeds , and its gradient fades to 0 near duplicates. Same goal, stable gradients.

KoLeo’s push explodes for near-duplicate tiles. KDE’s push stays finite.

CHIEF · instance-level supervision

Attention ranks patches. The highest- and lowest-attention patches supply binary pseudo-labels.

Generate pseudo-labels

Let a contain the patch attention scores. Select K patches from each end of the ranking.

P+=TopK(a)P−=BottomK(a)y~i={1i∈P+0i∈P−ignoredotherwise

Patches between these two sets are excluded from the instance-level loss.

Example · K = 2

PatchAttentionPseudo-label
A0.401
B0.251
C0.15Ignored
D0.10Ignored
E0.060
F0.040

Train an instance classifier

A binary head maps the patch representation hᵢ to probabilities for the two pseudo-label classes.

pi=softmax(Winsthi+b)

Illustrative binary cross-entropy formulation using the selected patches

ℒinst=−1|P+|+|P−|∑i∈P+∪P−[y~ilogpi(1)+(1−y~i)logpi(0)]

Labels 1 and 0 indicate high- and low-attention evidence for the slide category.

Treatment-conditioned residual predictor

The treatment and time gap condition how the patient representation changes.

Feature-wise Linear Modulation

FiLM uses conditioning information to scale and shift the MLP’s intermediate features.

h′=γ(c)⊙h+β(c)
h
Intermediate features from the MLP.
c
Conditioning information from the treatment episode and time gap.
γ(c)
Learned feature-wise scaling.
β(c)
Learned feature-wise shifting.

The same patient representation can be transformed differently for different treatments and time intervals.

Why a residual MLP?

The model predicts a change in the patient embedding, then adds that change to the starting representation.

z^t2=zt1+Δz
Patient embedding
z at t₁
Predicted change
Δz
Future embedding
ẑ at t₂

The FiLM-conditioned MLP generates Δz using the patient history, treatment episode, and time interval.

The residual connection carries the starting patient representation forward while the MLP learns the predicted change.

Understanding NT-Xent

One anchor i, its positive view j, and the other views in a batch of N images.

ℓi,j=−logexp(sim(zi,zj)/τ)∑k=12N𝟙[k≠i]exp(sim(zi,zk)/τ)

Increase the fraction. It assigns probability to the positive view j among all 2N−1 candidate views. The denominator includes the positive and 2N−2 negatives. The anchor itself is excluded.

Positive score ÷ all candidate scores → probability of selecting view j
Positive-pair probability pℓi,j = −log(p)
Low probabilityHigh probability

p is the positive-pair probability in the fraction above.

Pair loss −log(p)0.431

Why use a projection head?

The MLP separates learning useful features from optimizing the contrastive loss.

Image xEncoder fhMLP gzContrastive loss

Representation space h

The encoder produces a rich representation of morphology, texture, color, and other image features.

These features can support a range of downstream tasks, including tasks that use information affected by the training augmentations.

Contrastive space z

The projection head transforms h into z, where the contrastive loss is applied.

The loss encourages agreement between augmented views. This can suppress augmentation-specific information, such as stain variations.

A buffer between the objectives

Applying the loss to z gives the projector room to specialize in augmentation invariance, while h can preserve additional information useful downstream.

Gradients still flow through the projector to train the encoder.

After pretraining

SimCLR found that representations before the projection head, h, performed better on downstream evaluation. Discard the projector and retain the encoder for downstream use.

DINO intuition

Why centering and sharpening?

With no labels, the student can match the teacher trivially by giving every image the same answer. Each operation blocks one way of collapsing.

Centering: subtract a running mean

is the average teacher output over recent batches. Subtracting it means no single dimension can win for every image.

On its own, centering pushes the teacher toward a flat, uniform output. A flat target tells the student nothing.

Sharpening: a lower teacher temperature

Dividing by a small makes the teacher’s softmax peaked and confident, sharper than the student’s ().

On its own, sharpening lets one dimension win for every image, so every image looks the same.

Together: confident and diverse

Centering keeps the dimensions balanced across the batch. Sharpening keeps each image’s target confident.

The two pull in opposite directions, so the teacher can’t settle into either collapse.

Each patch gets its own clear target, so the student has something real to learn.

Centering stops one dimension from dominating. Sharpening stops the output from going flat. You need both.