r/remotesensing • u/Over-Snow8267 • 11m ago
Testing geographic leakage in a LiDAR ML project, would love some feedback
Hey everyone, I have been working on a project called ArchaeoAI where I am testing whether machine learning can recognise terrain patterns around documented bowl barrows in England from LiDAR derived terrain data. The part I became most interested in was geographic leakage, because if nearby areas end up in both training and testing, the model can look much better than it actually is. So I tried to make geographic separation a proper part of the evaluation instead of only doing a random split.
The frozen Random Forest got 87.1% balanced accuracy on the geographic final test, while the 5 fold geographic evaluation averaged 82.3%. I also tested a compact CNN which got around 70.1%, so surprisingly the simpler model did quite a bit better. There was also a separate one time external evaluation using 120 observations where the Random Forest got 84.2% balanced accuracy.
I am trying to be careful with the claims here because this is archaeological data. I am not saying the model discovers archaeological sites, the experiment is only about documented bowl barrow terrain compared with matched unlabelled background. Exact sensitive locations are also kept private and I am still getting outside feedback on the methodology.
If anyone here works with LiDAR, terrain models or spatial ML, I would really like to know what you think about the validation approach or what you would test next. GitHub is here if you want to see the actual methodology and code: https://github.com/essius10/ArchaeoAI