You want to model:

p(cluster =c | img)

p(c1 == c2 | dist(c1, c2) = d, img1 in c1, img2 in c2)

You could try a couple things:

• Frechet Inception Distance but instead of Inception model you use the medical CNN activations

• distance metric learning

• hdbscan/umap/etc for clustering

• persistent homology based topological data analysis methods for finding clusters

• masked autoencoders for good feature extraction

• JEPA style architecture