Hydroinformatics Blog - Monitor My Watershed Helps Lower Barriers for Real-time Data Sharing
Posted Jan 10, 2023
Hydroinformatics Blog Post
Organized by the CUAHSI Informatics Standing Committee. Contributions are welcome, please contact Veronica Sosa Gonzalez at email hidden; JavaScript is required.
By: Scott H. Ensign
There is growing demand from academic, agency, and community scientists for real-time data that is findable, accessible, interoperable, and reusable (FAIR). While the number of online environmental data portals continues to grow, the options for real-time data telemetry remain limited for scientists who value FAIR data. Hardware manufacturers offer proprietary remote communication products and data services like NexSens’ WQData LIVE, In-Situ’s HydroVu, YSI’s HydroSphere, Temboo’s Kosmos, and StormSensor’s Terrapin, but there are varying degrees of difficulty in making data from these services available to hubs in the internet of water (like CUAHSI).
In the do-it-yourself hardware space, makers can program cellular modems and long range radios to send data to Internet of Things services such as Ubidots, Particle.io, and ThingSpeak, but these data endpoints aren’t designed for data distribution or to serve as interoperable data repositories. Monitor My Watershed, Dendra, and Chords are a few open source data repositories capable of receiving real-time sensor data and contributing directly to hubs in the internet of water. The Monitor My Watershed Data Sharing Portal builds on a long history of collaborative efforts of the CUAHSI community to build a hydroinformatics data management tool. This article presents a brief history, current status, and future development of the Monitor My Watershed Data Sharing Portal.
The Observations Data Model (ODM) was developed in the mid-2000s to serve the diverse needs of the hydrological science communities of CUAHSI, and other initiatives for storing, managing, and sharing data and with prescient focus on linking unique identifiers of measurements within a hydrologic network (Horsburgh et al 2008). CUAHSI’s older HydroDesktop (no longer supported) and current HydroClient are based on ODM, and additional Python tools have been developed for ODM to visualize and post process time series data (Horsburgh et al 2015). This later tool set was later used to examine subjectivity in quality control processes on time series data (Jones et al 2018).
Over the next decade, the growing demand to accommodate ex situ sample data as well as in situ sensor measurements from diverse scientific fields, the need to accommodate ever-larger datasets including real-time data, and new standards developed by the World Meteorological Organization and the Open Geospatial Consortium inspired the development of ODM2 (Horsburgh et al 2016).
CUAHSI adopted ODM2 for storing time series datasets in its HydroShare service, and Hsu and colleagues (2017) demonstrated its application in four diverse Earth science disciplines. In 2019, Horsburgh and colleagues reported the development of an ODM2 web-based data sharing portal and its implementation as the Monitor My Watershed Data Sharing Portal. Monitor My Watershed began serving users in 2016 and was designed to accommodate both citizen science and academic researchers; the first large-scale use of Monitor My Watershed was by the Delaware River Watershed Initiative. Data within Monitor My Watershed is cataloged in CUAHSI’s Hydrologic Information System (a hub in the internet of water) and made accessible using CUAHSI’s WaterOneFlow API.
Monitor My Watershed uses ODM2 with TimescaleDB, an open-source timeseries database that supports standard SQL queries and is an extension of the PostgreSQL relational database. PostgreSQL is an open-source object-relational database that uses and extends the SQL language. This combination supports the extensive metadata allowed within ODM2 and preserves performance requirements as cardinality grows with additional sites, parameters, and related results.
Over 100 organizations and 157 users have contributed >220 million measurements from >1,400 sensors to Monitor My Watershed. Of the organizations using Monitor My Watershed, 49% are non-government organizations (NGOs), 24% are universities, 11% are agencies, 9% are municipal entities or corporations, and 6% are high schools. Most of the NGOs are watershed-based organizations conducting water monitoring and community-driven science.
Many users of Monitor My Watershed have been introduced to the tool through EnviroDIY, a community for do-it-yourself environmental science and monitoring maintained by Stroud Water Research Center. EnviroDIY has attracted a community of over 650 citizen scientists from NGOs, academic researchers and agencies, and municipal water agency staff and outreach specialists. This community has adopted data transmission protocols for sharing data through Monitor My Watershed, thus making data available to CUAHSI. Extensive resources are available at https://EnviroDIY.org to help beginning, intermediate, and advanced users program data loggers to send data to Monitor My Watershed.
Monitor My Watershed allows data upload and download as csv files. Data upload is particularly useful for filling gaps resulting from lapses in telemetry caused by a device’s power limitations or broader network outages. Data can be downloaded with a button on the user interface or by constructing a URL from the site and parameters’ universally unique identifiers.
The main page visualizes data sparkline plots of the most recent 72 hours of data and links to an interactive Time Series Visualization tool. With the latter, multiple parameters and sites can be added to a user-selected time frame.
ODM2 and its implementation in Monitor My Watershed are actively and openly developed (https://github.com/ODM2/ODM2DataSharingPortal); the most recent advancement to the web app was migrating it to Amazon Web Services. Currently, Monitor My Watershed is only leveraging a fraction of the power of ODM2 for managing environmental data. Monitor My Watershed will soon begin accepting observational data, such as bacteria and nutrient concentrations from grab samples and individual sensor measurements useful for quality control. Alongside these observational records, future updates to Monitor My Watershed may include enhanced metadata management tools for in situ timeseries and ex situobservations. We plan to enhance the Time Series Visualization tool with statistical summaries and plotting functionality. Finally, we hope to work with CUAHSI in implementing new APIs for better interoperability, providing an alternative to the relatively outdated WaterOneFlow API.
In summary, Monitor My Watershed serves people and organizations across a range of applications with a data portal for visualizing and managing a wide range of environmental data including real-time sensor measurements. Watershed organizations constitute almost half the user base (largely because of the Delaware River Watershed Initiative’s use of the tool in community science efforts). The Stroud Center developed extensive training resources for pairing DIY hardware with Monitor My Watershed, and this support has accelerated the growth and uptake of Monitor My Watershed across the US. Ongoing development of this hydroinformatics tool in coordination with CUAHSI serves the diverse needs of the CUAHSI community and beyond in contributing to the internet of water.
Acknowledgments
The ongoing and recent development of Monitor My Watershed is funded by Stroud Water Research Center and LimnoTech with support from the William Penn Foundation, American Water, Brookby Foundation, and Amazon Web Services.
Portions of this article were previously published on 17 November 2022 in the Water Features blog of the Internet of Water Coalition website:
https://internetofwater.org/bl...
About the author: Dr. Scott Ensign is an aquatic and estuarine scientist at Stroud Water Research Center, a non-profit affiliate member of CUAHSI.
References
Horsburgh, J. S., D. G. Tarboton, D. R. Maidment, and I. Zaslavsky. 2008. A relational model for environmental and water resources data: Observations Data Model. Water Resources Research 44.
Horsburgh, J. S., S. L. Reeder, A. S. Jones, and J. Meline. 2015. Open source software for visualization and quality control of continuous hydrologic and water quality sensor data. Environmental Modelling & Software 70:32–44.
Horsburgh, J. S., A. K. Aufdenkampe, E. Mayorga, K. A. Lehnert, L. Hsu, L. Song, A. S. Jones, S. G. Damiano, D. G. Tarboton, D. Valentine, I. Zaslavsky, and T. Whitenack. 2016. Observations Data Model 2: A community information model for spatially discrete Earth observations. Environmental Modelling & Software 79:55–74.
Horsburgh, J. S., J. Caraballo, M. Ramírez, A. K. Aufdenkampe, D. B. Arscott, and S. G. Damiano. 2019. Low-Cost, Open-Source, and Low-Power: But What to Do With the Data? Frontiers in Earth Science 7.
Hsu, L., E. Mayorga, J. S. Horsburgh, M. R. Carter, K. A. Lehnert, and S. L. Brantley. 2017. Enhancing Interoperability and Capabilities of Earth Science Data using the Observations Data Model 2 (ODM2). Data Science Journal 16:4.
Jones, A. S., J. S. Horsburgh, and D. P. Eiriksson. 2018. Assessing subjectivity in environmental sensor data post processing via a controlled experiment. Ecological Informatics 46:86–96.