Cod drumming is the sound Atlantic cod make during spawning. The sound is made by muscles attached to the swim bladder, which make the swim bladder vibrate and produce a low grunt. Since the grunts are linked to spawning, recordings of them can show when and where cod spawn. However, the recordings also contain many other sounds. This means that the sounds first have to be found and classified, so that the cod grunts can be told apart from the rest.
Doing this by hand takes a lot of time. An underwater microphone, called a hydrophone, can record for weeks without a break, and listening through all of it is not realistic. Biologists at UiA have recorded these cod sounds and marked thousands of them by hand. This work was done through UiA’s scheme Studenter i innovasjons- og forskningsprosjekt (students in innovation and research projects), as part of a collaboration between the Centre for Artificial Intelligence Research and biologists at UiA. The aim was to find out whether artificial intelligence can find and classify the sounds automatically. Five kinds of sounds were used, namely cod grunts, very short clicks, the sound of moving water such as bubbles, silence, and other sounds.
Teaching a computer to recognise sounds from scratch requires a large number of examples. For this reason, this project used Perch 2.0, a freely available model made by Google DeepMind. Perch was trained on a very large collection of animal sounds, mostly bird song, but also frogs, insects, and mammals. As a result, it has learned what makes animal sounds different from each other. Perch describes every sound as a long list of numbers, called an embedding. The embedding works as a fingerprint of the sound, where similar sounds get similar embeddings. For example, two cod grunts get embeddings that lie close to each other, while a grunt and the sound of bubbles lie further apart. This means that only a small and simple model has to be trained on top of the embeddings to tell the cod sounds apart, which needs far fewer examples than training a model from scratch.
The method had two parts. Firstly, Perch was tested on almost 30,000 short sound clips that the biologists had already cut out and labelled, where the test clips were never used for training. Secondly, the model was tested on full recordings, which is how it would be used in practice. This was less straightforward. Perch analyses five seconds of sound at a time, and it makes one embedding that sums up the whole five-second piece. A cod grunt, however, lasts only a fraction of a second, and a click only a few thousandths of a second. In other words, a short sound makes up only a tiny part of each piece, and the embedding mostly describes the background around it. To solve this, a simple detector was added in front of Perch. The detector finds the moments where something louder than the background happens, and each of these moments then gets its own embedding.
On the short clips, the model labelled 92% of the test clips correctly. It recognised 97% of the cod grunts, 95% of the clicks, and 92% of the silence. Water and other sounds were harder, with roughly half of them labelled correctly, most likely because these two groups vary a lot. On ten full recordings that the model had not seen before, about four hours of sound in total, the system found and correctly labelled 77% of the grunts and 93% of the clicks. This shows that most of the cod sounds can be found automatically, also in new recordings.
However, the system also reported many more sounds than the biologists had marked. Some of these are noise, but some may be real cod sounds that were never marked, since not every sound in the recordings is labelled. To reduce the number of false alarms, the model was also taught a sixth group, background, from moments where the detector reacted but no sound had been marked. This reduced the number of reported sounds to about one seventh, while the share of grunts and clicks found fell to 71% and 87%. This is a trade-off. A tool where a biologist checks the results should find as much as possible, while a tool that counts sounds automatically should make fewer mistakes.
For this project, I was involved in every step of the work. This included (1) preparing the raw recordings and the marked sounds, (2) setting up Perch and training the model on top of it, (3) building the detector for the full recordings, and (4) testing the system and presenting the results to the biologists. One important lesson was that real data is rarely as clean as expected. For example, sounds can be marked slightly too late, and not every sound in a recording is marked. In addition, working with biologists gave valuable experience in explaining technical results to people from another field.
In conclusion, a general model for animal sounds, combined with a simple detector, can point biologists to most of the cod grunts and clicks in hours of underwater recordings. Since Perch was never trained on fish, this suggests that what it learned from birds and other animals can be reused for sounds underwater. The system cannot yet replace a trained ear, as water and other sounds remain difficult, and many of the reported sounds still need to be checked.
In future work, other deep neural networks could be tested on the same recordings to see whether a model trained directly on the cod sounds performs better than Perch. In addition, the system could be tested on recordings where every sound has been marked.