Group

Data management and processing tools / Feed

Conservation tech work doesn't stop after data is collected in the field. Equally as important to success is navigating data management and processing tools. For the many community members who deal with enormous datasets, this group will be an invaluable resource to trade advice, discuss workflows and tools, and share what works for you.

discussion

Need advice on data for an app that recommends plants.

This is going to show how little I know, but I'm doing my best! Basically I could use help understanding what I should do from a data standpoint in order to enable a...

1 2

It's a really good question, Colleen!

Ideally, I would work with two developers, or at least one developer and one other party who knows what developing an app like this would mean. This will lower the risk that the developer answers your question too much in their advantage. So this is one reason why it is a good question, and maybe this is why you ask it. However, if you really trust your developer not to take advantage, then go with just her/him/they.

It's also a good question because there is no easiest way to go about this if you're on a budget. You mention that you do not have a tech background, so here comes some explaining. If I misunderstood, then please skip and continue at "Back to 'the easiest way to go about this'"

There are at least two things you should be really aware of when it comes to software development.

The first is that once the basic data structure is defined and the software built around it, it is extremely costly to change the data structure. It's like deciding after the car has been built, that the engine should go to the back of the car instead of the front.

The second thing is that the basic data structure is dependent on ( among other things, but I'd say these are the two most important factors ) the complexity of what needs to be achieved and the speed at which it needs to be done.

The difficulty is that the required complexity and the speed may change over time, which brings one back to the car and engine situation. Changes in complexity and speed requirements may be the result of many things, one of which is success. You get far more clients than anticipated, so the system needs to be scaled up. In addition, with more users come more feature requests ( this can work both ways: new features result in more users, and more users may result in more feature requests ).

There is no real solution to this problem ( well, except not growing beyond the point that the first design can handle ). When it comes to scaling up, one vendor may claim that their database back bone easily scales up. Maybe so - but it may come at a price and they may also underestimate your and their own future needs. When it comes to changes in complexity, additional features can in the beginning probably be added on without changing the basic data structure. Maybe an additional row of seats at the back of the car, a trailer hook, bigger lamps, a roof-rack, a trailer, suitcases on the rack. At some point the car will need a new and bigger engine to carry all those add ons and keep at the same speed.

Here is a prediction : the more you stress cheap and efficient at the beginning, the bigger these problems will be later on. But when the business is successful, there will be more money to invest in scaling up and redesigning. Obviously yes, but in terms of the car metaphor, you may find that you want the car to be kept running with all its added on features, while the engine is replaced and moved to the back. It may be possible, but perhaps out of reach of patience and the increased income.

 

Back to 'the easiest way to go about this':

Invest a little effort to find out not only what are the minimum requirements for the MVP, but also what else you or your clients may want in the future. The developer should then have these future requirements in mind (and future upscaling) when they start developing for the minimum ones. This means, develop a somewhat more generic data structure than what is necessary for the MVP. This will cost some more at the beginning but should save a lot later on. I'm writing 'should' not 'will' on purpose. It's a balancing act because taking too much into account has the risk of over-engineering for a future that may not happen, or develop differently than expected. Like I said, there is no easiest way out.

 

A few more detailed comments

If there is no API for a source, try to go around it if possible at the beginning. API's are made with some long term stability in mind. Websites and web pages not necessarily so or less so. They will require more monitoring and maintenance on the web scraping routines.

Perhaps the developer may not be aware of the necessary pre-work that needs to be done if the pre-work depends on biological knowledge needed to transform the data from your data sources into data that allows easy ( and fast ) calculation of results to the users. 
 

See full post
discussion

Automatic extraction of temperature/moon phase from camera trap video

Hey everyone, I'm currently trying to automate the annotation process for some camera trap videos by extracting metadata from the files (mp4 format). I've been tasked to try...

10 1

I just noticed that TrapTagger has integrated AI reading of timestamps for videos. I haven't had a chance to try it out yet, but it sounds promising. 

Small update. I uploaded >1000 videos from a spypoint flex camera and TrapTagger worked really well. Another program that I'm currently interested in is Timelapse which uses the file creation date/time. I haven't yet tried it, but it looks promising as well. 

Hi Lucy,



I now realised it is an old thread and you most likely have already found a solution long ago but this might be of interest to others.

As mentioned previously, it is definitely much better to take moon phase from the date and location. While moon phase in general is not a good proxy for illumination, that moon phase symbol on the video is even worse as it generalises the moon cycle into a few discreet categories. For calculating moon phase you can use suncalc package in R but if you want a deeper look and more detailed proxy for moonlight intensity, I wrote a paper on it 

 with accompanying R package called moonlit 

When it comes to temperature I also agree that what is recorded in the camera is often very inconsistent so unless you have multiple cameras to average your measurements you are probably better off using something like NCEP/NCAR Reanalysis (again, there is an R package for that) but if you insist on extracting temperature from the picture, I tried it using tesseract and wrote a description here: 

 

Good luck!

See full post
discussion

How do you fund or finance your long-term data infrastructure?

Today at the Bioacoustics Data Networks and Platforms webinar during the after hour, there was a great discussion about long term data infrastructure. Not just how to...

6 2

Hej @kimhendrikse,

Not really the kind of answer I was looking for ( because we now still don't know how you finance your setup ), but it is an interesting answer to the question which is also practiced here and there. But the financing problem will not disappear. It will just be distributed as well. The nodes still need funding. Less so per node, which might make it a lot easier, but there is still funding involved. Perhaps there is even a way to distribute the 'central' catalogue. I wouldn't insert a central repositoring chaching data because then we're back at something big and expensive again. 

Also, there will be costs involved in developing and maintaining standards, which also need to be paid - through membership fees perhaps. Lastly one would need a way to deal with those nodes that cannot fund themselves anymore. It would lead to fewer and bigger nodes and should not happen too much because then we're back at square one.

 

"because we now still don't know how you finance your setup"

I pay for everything myself out of the salary from my full time job :-(

But... long term I'm seeing a future in starting a company that sells products. (Maybe not that long term).

That's a really honest answer Kim! Thank you for clarifying

You're likely not the only one. In fact it would be interesting to find out how much conservation tech use and development is funded through personal means. I made a note. Perhaps we'll do a poll sometime

See full post
event

Design-led innovation for nature

How can space-enabled data and services be translated into information that is trusted and actionable, fuelling responsible behaviours towards nature? A new programme where SMEs will be paired with design consultancies...

0
See full post
discussion

efficient methods for labeling clusters of fish in images for segmentation purposes 

Hi everyone,Our team is working on a project involving the segmentation of individual fish within clusters (groupers specifically). We have images with hundreds of groupers...

2 0

hi, if your intention is to use programming and a programming language like Python then one option which can give good results is the SAM (Segment Anything) or SAM-2 algorithm. The following weblinks include more information:

 

Agreed. I have recently begun using SegmentAnything as a replacement for Detectron and have been very pleased. However, for a fish school I don't know how well it would do out of the box with the many overlapping fish. 

See full post
discussion

Rapid Camera Trap labeling with ChatGPT

Hi all, I've published a blog post on rapidly labeling camera trap data using ChatGPT for species identification and a simple object detection model to get the bounding boxes....

3 6

super interesting, thank you for sharing! definitely will watch!

Really interesting, I will take a look

See full post
discussion

WILDLABS AWARDS 2024 - Mobilizing Motus: optimizing a wildlife tracking data pipeline for aninternational, collaborative conservation technology community

We’re delighted to introduce our project, Mobilizing Motus. Motus, a Latin word for movement, is a collaborative wildlife tracking system that uses automated radio telemetry...

2 4
See full post
discussion

Firetail 12.2 released

Hi all!I'm happy to announce that we just released Firetail - release 12.2A few new things in a nutshellA new evaluation module for evaluating the performance of predicted...

2 2
discussion

Behavioral logging from video

Hi everyone.  I wanted to ask if anyone has found a tool that they'd recommend for logging/coding behavior from video.  I've used BORIS and do like that (and it's free...

2 2

Hi @KylerA long time no speak! I've used BORIS mostly too, but I've also heard good things about CowLog, so could be worth a look. I also found this (now 4-year-old) paper that discusses a few different options, in case you hadn't seen it before. 

All the best,

Rob

See full post
discussion

Simple Data Standards for wildlife reserve management scenarios?

Hi - I've been lurking on Wildlabs for a while, consuming as much info as humanly possible, but I seem to miss specific info for my current needs.I'm the self appointed in house...

2 1

Just as a note, you might also be interested in specific protected area management software like SMART or EarthRanger as well!

Thanks!

I'm definitely looking at Smart and EarthRanger for overall management eventually, but for now I just want to track what we have better - especially as a base map to support further advanced management activities. Funding and available human resources to effectively manage a tech based scientific and best practice focused solution is in short supply - but will hopefully be better addressed within the next year or two. 

 

See full post
discussion

Auto ID Software for NA Amphibians?

Hi All!So I'm currently running 4 Wildlife Acoustic Song Meter MiniBats at my beaver research and control sites as a means to measure species richness. 2 units at each site, one...

2 0

Hi Cortney,

I recommend exploring the tools developed by Kitzes Lab, such as OpenSoundscape, which can be used for the automatic detection of anurans. Here's a reference paper that might be useful. You might also consider reaching out directly to Justin Kitzes @jkitzes  for more specialized advice.
Also here is a another paper that use BirdNet to automatically detect the bullfrog presence.
Best of luck with your analyses!

Retweet on OpenSoundscape - great package and documentation that allows you to build your own CNNs! Note that this is in Python though. 

There are tons of bioacoustics software and packages in @tessa_rhinehart 's helpful list too! Some more recent options that I don't think are on that yet are Whombat and gibbonNetR (which uses the 'Torch for R' ecosystem). 

Also, Raven has template- and energy-based detectors you could try as well. The monitoR and warbleR packages allow you to do template-based detections too. BirdNET is integrated into Raven now as well.

You can also try Kaleidoscope's native clustering function.

Agree with you that there's definitely not enough resources for frog detection though! I'm assuming the Australian one you found is FrogID? There is also AnuraSet, but this is for neotropical frogs.  

See full post
discussion

Move BON Development: Follow up discussion

Hey Biologging Community! We just launched a new initiative to mobilize animal tracking data in support of national and global scale conservation goals (learn more here!). If you...

6 6

Hi Talia! 

I feel like the topic is so broad that it might help to put some constraints around things, see what works, and then broaden those out. I have a lot of ideas regarding the data monitoring and collection side based on the other sensor and observation networks we've set up in the past. 

There may also be some potential scope to incorporate things like data collection and integrated monitoring to the Build Your Own Datalogger series where the system is updated to feed data into the observation network. 

It'd probably take a bit of discussion and coordination. Let me know if interested. I'm fine to jump on a call or discuss via email too.

@cmwainaina please take a look

See full post
discussion

Looking for literature materials/any useful data on HHC(Human-Hippopotamus Conflict) mitigation and coexistence.

Hello Wildlabbers,I am looking for anyone who have done some research on human-hippopotamus conflict mitigation and coexistence or anyone who might have some scientific info/...

8 7

Hi loveness,

Your doing great here are some research that may be helpful on what your looking for

search for article of Human-Hippopotamus Conflict: Impacts and Mitigation Strategies published in the Journal of Wildlife Management.

https://www.google.com/search?client=ubuntu-sn&sca_esv=23430e1df8f38b82&channel=fs&q=Assessing+Human-Wildlife+Conflict+with+Hippopotamuses+in+the+Context+of+Wildlife+Conservation+from+the+African+Journal+of+Ecology.&sa=X&ved=2ahUKEwigkNqHx5SHAxXi0gIHHaOBA-EQgwN6BAgFEAE

 

 

See full post
Link

Rapid Camera Trap labeling with ChatGPT

Hi all, I've published a blog post on rapidly labeling camera trap data using ChatGPT for species identification and a simple object detection model to get the bounding boxes. While presented in Edge Impulse, this approach can be replicated outside the platform using Python...

1
careers

Data Manager, Conservation International

Conservation International are seeking a data expert to help (a) serve as a day-to-day technical project manager to ensure data collection, data management, and data quality control with global program teams and...

2
See full post