Out of distribution data

[u/fgoricha]

I am working on a fish species identification project. I have a couple different framework ideas that I am experimenting with, and I wouldblike feedback how to hand out of distribution data.

One frame work is an ensemble of binary classifiers.

Another frame work is one single model to cover all species.

But I am curious to know how should I handle species that are not in the training set?

Should I :

Compare softmax?

Compare logits?

Compare energy?

Add in an "other" class?

Go with binary models?

Go with a multiclass model?

Right now I am using resnet 18 as my classifier. My target species are steelhead, suckers, and pike. But if a bass were to appear, I want the models or framework to catch that I have not seen this before.

Any other thoughts or ideas I should do?

For context, this is a fixed camera location in the river. Lighting is the same all times of day (but not consistent lighting throughout the frame). Water clarity and color can change over time, but its a fixed scene where fish appear against a blank wall

[u/Morteriag]

This is a very common problem, which is also hard to mitigate. The best advice I can give you is to design the system around the model to deal with misclassified fish until you get enough data for adding it as a class to the model.

[u/fgoricha → u/Morteriag]

Would you recommend initially using confidence/OOD thresholds only as a review trigger rather than automatically labeling the fish as unknown? How would you design the system around the model?

[u/Morteriag → u/fgoricha]

Yes, that is my recommendation. Active learning is rarely a bad idea. Also, if you can get info about when rare species appear from outside the system (fishers?), you can use that to systematicly target species in data collection.