Technology
Components
The Cacophony Project is built with a wide variety of technologies including embedded systems, web applications, backend services and machine learning pipelines. Let's take a look at the various components of the project in a little more detail.
Remember, the project is completely open source. This means that anyone can customise the functionality of their devices, and has permission to reuse or modify any of our source code for their own needs.
Thermal Camera + Bird Monitor
Equipped with a thermal sensor, microphone, and (optionally) external speakers, it is deployed into New Zealand's native bush to automatically lure and identify invasive predators. The Thermal Camera software is a combination of Rust, Go and Python code. A version of this device without the thermal sensor is available as a stand-alone bird monitor.
| Base hardware |
Raspberry Pi 3 RP2040 microcontroller |
| Thermal camera | FLIR Lepton 3 |
| Additional hardware |
Custom interface hat ATtiny microcontroller 4G modem Real time clock MEMS microphone 256MB embedded NAND flash |
| Operating System | Raspberry Pi OS |
| Key Software |
tc2-firmware tc2-agent thermal-recorder thermal-uploader |
Sidekick mobile app
The Sidekick companion app is available on both iOS and Android.
It’s the missing link between our Thermal Camera and Bird Monitor hardware, and those last-mile, hard-to-reach locations – far beyond the reaches of cellphone towers.
Sidekick allows staff in the field to visit deployed devices and collect monitoring data via from our Thermal Cameras and Bird Monitors via WIFI. No messing about with SDCards, no opening up devices and potentially compromising their weather sealing. When you’re back in coverage, Sidekick can upload the data to the Cacophony Monitoring platform for processing and analysis.
Sidekick is also the interface through which you configure any of our devices. Set the times you’d like it to wake up and record. Synchronise the internal clock, and set the GPS location attached to the monitoring data.
Sidekick is built using Kotlin and various web technologies.
API Server
In many ways, the API server is the heart of the project. It's where all video and audio recordings for the project are stored, processed and then made available for later querying and download. All the components of the project interact with the API server in one way or another.
| Platform | Node.js |
| Database | PostgreSQL |
| Object Store | Minio (or other S3 compatible service) |
| Test infrastructure |
Implemented with Cypress Continuous integration via Travis |
| Source code | cacophony-web |
Web Application
The Cacophony Project has a full-featured JavaScript web application built using VueJS 3 for querying, tagging and managing uploaded recordings. It interacts exclusively with the Cacophony API server.
Machine Learning
Automatic classification of predators in a thermal video feed is a crucial part of the project. This magic is implemented using a TensorFlow based machine learning model which is trained by our classifier pipeline. We make heavy use of a number of data science tools including NumPy, SciPy, OpenCV and HDF5.
If you're into data science or machine learning, maybe you could help us improve our classifier's accuracy and efficiency.