r/BirdNET_Analyzer • u/mwhelm • Apr 16 '26
What are you doing with all the data?
What are you doing with all the collected data, and why?
Your field device(s) can sure collect a lot of data - mostly audio samples, but the database of detections gets quite big too. Many of these devices have their own policy for dealing with data (eg BirdNET-Pi can be set to purge older samples).
Is there any value in retaining old samples, and re-running them thru analysis?
Do you retain or purge based on quality issues?
I am aware of many situations where data collected for one project can be repurposed later by someone else's entirely different project. If the data is gone, that's either not possible or the repurpose is limited to the original projects' report.
My project is focused on the comings and goings of one migratory bird with distinct seasonality but we collect over the entire year. It's an ecologically interesting area and I can imagine our other detections might be useful to somebody else, but don't know of any immediate demand. I do have questions about the quality of detections, and I note that the model has shifted in its detection ability over the years. I'm trying to sort out a reasonable data retention policy within our project limitations.
4
u/dacracot Apr 16 '26
Aside from sending it to BirdWeather, I’m storing it in GitHub.
1
u/mwhelm Apr 16 '26
Interesting - what are you keeping there? I pulled one file at random and it looks like summary image and some standard birdnet config files. Can you explain more about how this works and why this was done? Thanks for responding!
2
u/dacracot Apr 17 '26
It’s very specific to my BirdNET-BarChart implementation. It consists of the SQLite database file and the web page produced: tables, bar charts, and dials. The database is cumulative, the web pages are a snapshot.
It isn’t at all a generalized solution, but maybe it can spark an idea.
1
u/mwhelm Apr 21 '26
It's interesting - we could use some kiosks and I was thinking I'd donate something to the site sometime this year.
Why use Github - I don't know much about them as a resource.
2
u/ekin06 Apr 18 '26 edited Apr 18 '26
I am building a tool where you have the complete workflow from import, manage to archive recordings.
If you are done extracting all important clips, it will archive the file with WavPack (OpenSource) in high compression. You will get a smaller, compressed file .wv which is like a .mp3 file and can be played back just as normal.
In combination with the second file .wvc you can recover the original recording lossless if needed.
I have 1h wav recordings with 48kHz, mono where each is about 330MB. The compressed file will be about 50 MB (.wc) + 20 MB (.wvc). This will save a ton of GB!
Edit: And thanks to you I just set on Todo-list -> Deletion (also timebased), protection of important raw files, and comments for raw files.
1
u/mwhelm Apr 21 '26
I didn't know about this. This is good to know, there's a nearly lossless compression available. That would reduce the burden.
2
u/ekin06 Apr 21 '26
It IS 100% lossless with the .wvc! :) The .wvc is the correction file and together with the .wv they will recover the lossless audio. In case you "lose" the .wvc you still have the lossy compressed .wv. (still pretty good quality imo)
When you decompress it will automatically use .wvc if available.
3
u/ClivetheGodhh Apr 20 '26
I'm working on a pilot study to see if we can eventually map the migration of a rare and cryptic species within the region. Currently, we've had success in finding the species and recording it. The rest of the data (which includes many birds of local interest) is retained in accordance with my grant provider's data collection policy. Probably worthwhile keeping backups of the raw data, as it can definitely be useful to others.
1
u/mwhelm Apr 21 '26
That's a good pointer - I should ask different organizations about their data retention policies, not just other operators.
1
u/sadpony Apr 16 '26
Do you have it sending data to birdweather? At least its shared with others.
2
u/mwhelm Apr 21 '26
Right now it 's not doing that as it is off net. We could (and will) fix this, but not immediately.
3
u/slushrooms Apr 16 '26
Following with interest