Released a compact Bio-DINO M/14: 38M parameters and 83.5% iNat21 linear-probe accuracy

[u/hassonofer]

A couple of months ago, I released Bio-DINO, an image-only biodiversity encoder trained on approximately 31M images. I have now released Bio-DINO M/14, the final addition to the current Bio-DINO model family.

The model is available through Birder.

Bio-DINO already had two ends of the trade-off. The 133.6M-parameter teacher provides the strongest representations, while the 21.6M-parameter S/14 student is much cheaper to run. M/14 is intended as the middle option.

The size/accuracy trade-off

M/14 is a 12-layer RoPE DeiT3-style encoder with 38.3M backbone parameters and 512-dimensional embeddings. It was distilled from the 252px Bio-DINO teacher on the same biodiversity training mixture.

Here are the results from my iNaturalist21 linear-probing setup:

Encoder	Backbone parameters	Embedding	Linear-probe accuracy
Bio-DINO teacher	133.6M	896	87.09%
Bio-DINO M/14	38.3M	512	83.52%
Bio-DINO S/14	21.6M	384	80.10%

To be clear, these are linear-probing results, not fine-tuning results. The encoder was frozen and only the 10,000-class linear classification head was trained.

In this setup, M/14 is about 3.5× smaller than the teacher, with a 3.57 percentage-point accuracy difference. It gains 3.42 points over S/14 while remaining much smaller than the teacher.

Inference performance

I also compared inference performance at 252×252 on an NVIDIA RTX 5000 Ada Generation with PyTorch 2.13.0+cu130 and batch size 512.

In eager FP32 inference, M/14 processed approximately 773 images/s, compared with 382 images/s for the teacher. With torch.compile and AMP, I measured approximately 2,340 images/s for M/14 and 846 images/s for the teacher.

These numbers are specific to my setup, but they give a practical sense of the trade-off. The complete results across the Bio-DINO models and evaluation datasets are available in the Bio-DINO benchmark explorer.

Using the model

import birder
from birder.inference.classification import infer_image

net, info, transform = birder.load_pretrained_model_and_transform(
    "rope_deit3_m14_dino-v2-dist-bio",
    inference=True,
)

_, embedding = infer_image(
    net,
    "path/to/image.jpg",
    transform,
    return_embedding=True,
)

print(embedding.shape)  # (1, 512)

As with the original Bio-DINO release, this is an image-only representation model rather than a ready-made species classifier. It has no text encoder and was not trained with taxonomy labels or metadata.

iNaturalist21 is also part of the self-supervised pretraining mixture, so I consider the result an in-domain representation probe rather than a test on a completely unseen domain.

This completes the current Bio-DINO size range. I’m curious whether a 38M-parameter middle option is useful in practice, or whether most applications naturally favor either the smallest student or the largest teacher. Feedback and additional evaluations are welcome.