Dear Motus collaborators,
We want to provide some updates to the community about developments that are coming to the data processing system that supports Motus.
Starting in 2024, the Motus team at Birds Canada initiated a plan to fundamentally rewrite many of the components of the Motus data processing pipeline, with the goal of improving timely availability of data, user experience and supporting the long-term delivery of the network. While the existing Motus data processing software has sustained Motus for more than 12 years, it is nearing the limit of its capacity. For more information about how Motus data is processed see How Data are Processed | Motus Docs .
Receiver data uploaded to Motus is placed in a queue to be processed into tag detections. As of 2026, Motus has grown to the point that files are being added to this queue at about the same rate as they are processed and removed. This means that any downtime or seasonal peaks in data volume results in persistent delays that are difficult to clear. While no data is being lost, the delay between upload and processing is currently approximately 4-6 weeks. As many are aware, this backlog means delays for users waiting to receive their data, confirm station or tag performance. While we continue to upkeep this current system, we are aware that a rewrite to address scalability is overdue. We have secured funding to complete this rebuild, and it is currently underway.
We anticipate that the rebuild will lead to faster data availability for researchers, reduced transmission costs, improved data quality, and more efficient data handling and analysis on the server and within the R package. These foundational improvements to support greater data volumes will also allow the network to maintain and grow for years to come. We have separated our plans into two phases, each with its own timeline.
Phase 1: Replacing our legacy data processing system
We have completed a new algorithm used to convert radio pulses into tag detections. This software is called Burstfinder, and it will be used in two ways. First, when files containing radio pulse data are received by Motus, they will be processed through Burstfinder. Based on early results, this new algorithm appears to improve signal detection and leads to a reduction of missed tags and false positives. We are assessing the performance of Burstfinder under various conditions and will be reaching out to the Motus community to invite beta tester participation in the winter of 2026-27. Second, we are actively developing ways to accelerate data processing on the stations themselves which will significantly reduce the data volumes needing to be transmitted to Motus.
In addition, we are rewriting the software that processes the burst files and writes them to the Motus database. This new data flow will by-pass the legacy processing system that is responsible for most of the processing backlog, while eliminating the need to ever reprocess detection data due to changes in other system (e.g. tag) metadata. Together, these changes will result in reduced data processing volumes, data transfer time, and fewer steps during data upload and lead to significant improvements in processing time and scalability.
Once these components are operational, and we have gained sufficient confidence in the quality of the output through collaboration with the Motus community and beta testers, we will start gradually diverting data to this new pipeline for an increasing subset of receivers. Once we have full confidence in the new system, we will use it to process all data (historic and ongoing), except for a few specific use-cases that will continue to use the legacy system (e.g. data from SRX receivers). We anticipate these changes will not require Motus collaborators to change their operations, other than to optionally update the SensorGnome software on their receivers.
Our hope is that Phase 1 implementation will roll out to all users in 2027.
Phase 2: Optimizing data storage and delivery
Concurrent with our work on Phase 1, we are planning for Phase 2 which will optimize how we store and deliver data to users. Rather than store all data in a single SQL Server database (currently over 8 billion hits), we will transition to storing larger data tables in a column store format (likely Parquet); which is technology purpose-built for big data applications. This will lead to faster data queries and downloads reduced storage space needs, and improved scalability with the growth Motus network to support more stations, new technologies and new efficient ways to deliver large data volumes to users. We plan to be able to deliver data directly in a parquet format, which is directly readable in R with common packages. We anticipate Phase 2 to be implemented in late 2027 and will be inviting community feedback and asking for beta testers at that time.
A message from the Motus team:
We are humbled by the widescale growth and adoption of the Motus network and are consistently amazed at the innovative ways this community has used Motus to power research and conservation. We are working hard to refactor and scale the software underpinning Motus to keep pace with this growth and the needs of this community, ultimately with the goal of supporting the conservation of the species we deeply value.
We apologize for the delay in rolling out these necessary improvements. We understand that processing delays have caused some confusion and frustrations, and that there may be growing pains associated with this transition to an updated system. Ultimately, we hope the result will be a system that is more accessible and versatile, with greater capacity for growth and an underlying structure that can nimbly accommodate technological advances. We are committed to making these investments because we believe Motus is an essential tool for studying animal movement, facilitating collaboration, and informing conservation. We sincerely appreciate your patience during this period of transition.
Denis Lepage Senior Director, Data Science and Technology Birds Canada