Hi everyone,

I recently watched a talk by Aza Raskin where he discusses the idea of using generative models to explore communication with whales. While the conversation wasn’t deeply technical, it sparked a line of thought I’ve been quietly exploring: can we use generative AI to better understand, replicate, and eventually "translate" the communicative structures in animal vocalisations?
I work primarily in conservation and ecology, and I’ve applied several predictive models (CNNs, LSTMs, and some transformer-based architectures with transfer learning). However, I’ve never trained a generative model myself. Still, I’ve been sketching a conceptual pipeline that could link bioacoustics and behavioural ecology.
The idea is now something like start with a large, unlabeled dataset of an animal vocalisations (e.g. bird songs, whale calls, primate vocalisations). Use unsupervised learning (e.g. embeddings + UMAP + HDBSCAN or even kmens, don’t know) to group vocalisations by structure or spectral similarity.
Then, based on field knowledge or ethological observations, manually label some clusters with possible communicative functions (e.g. alarm, contact, courtship). Only if they make sense. Use these labelled clusters to fine-tune a generative model (like SpecGAN, AudioLDM, or even an autoregressive model like WaveNet or VALL-E) to create synthetic sounds conditioned on function.
But… then explore whether this can help us simulate or even engage in meaningful communicative loops, perhaps as a tool for playback experiments or for probing animal perception in a controlled field experiment or exsitu one.
This is still a very very very early-stage idea, more like a sketch hahaha. but I’m curious whether anyone in this community has tried something similar. Not necessarily for whales—any species or sound system would be relevant.