Environmental data that enables people to search and compare real time data on river levels and flows in a similar way the Trivago website enables people to search and compare the prices of hotel rooms is now available from the Ministry for the Environment website.
The Environmental Integrated Data Infrastructure or “e-IDI” makes environmental data more discoverable, shareable and accessible. The e-IDI was developed by the Ministry, Natural Resource Sector, regional government sector and research organisations. The project was funded by the Better Public Services Seed Fund, administered by the Treasury. The service builds on the National Environmental Monitoring Standards data exchange protocol (Environmental Observation Data Protocol) developed between NIWA, Landcare, GNS, and Horizons Regional Council.
Although work is underway to standardise data and processes, central and local government agencies and crown research institutes currently generally use manual, one-to-one processes to send, search and access data. The result is that each time data is shared amongst agencies, the data must be collated, cleaned and reformatted before even basic use or analysis of the data may begin. That takes a lot of time!
The e-IDI was created by applying a linked data approach which allows data from multiple sources and different systems to be brought together using a common set of standards and vocabularies.
What the e-IDI can be used for
There are many uses for e-IDI.
- Nationally accessible real-time linked data. This data could be used across NZ for disaster management (ie: flooding)
- Ability to filter, group and compare sites across regions and time series. This data can inform multiple monitoring and analysis decisions.
- Machine readable data to support use across various machine-to-machine platforms. The streamlining of accrual and manual processing has a number of financial and efficiency gains.
- Standardised definitions and vocabularies for ease of interpretation. This allows for greater interpretation and use of data in a standard way.
- It’s a project that’s at a proof of concept stage.
Water quantity data only at this stage
Water quantity data is being piloted because the data is operational and non-personal. Additionally, water quantity data was already standardised in real-time for the Land, Air, Water Aotearoa (LAWA) website in 2015, making it easier than other domains to test in a shared infrastructure.
What happens next?
e-IDI was funded just for a proof of concept and a business case, not for a pilot and for rolling out an operational system – though it’s clear that the concept of moving the natural resource system from manual data preparation and transfer, to direct digital search and access has been proven.
The Ministry for the Environment project lead ran workshops with ALGIM in early November to share what was learnt from the proof of concept. It will consider recommendations from the project and go forward from there.
Reports and other outputs
Reports are available in both PDF and markdown formats, in keeping with linked data’s human and machine readable requirements.
All the reports and other outputs are openly licensed, as the PoC’s main aim was learning and sharing the results. We have also used persistent identifiers for the reports.
- The e-IDI Proof of Concept Beta site, a brokering solution that transforms different data from disparate systems into a harmonised form for integration is here: http://envdatapoc.co.nz
Watch the walk-through video for the e-IDI Proof of Concept Beta site.
- The interactive river flow data explorer for the e-IDI was built with RShiny, view the interactive flow data
Watch the walk-through video for the RShiny application
The code for the RShiny application is here
- The summary report is a short, plain summary of the project and recommendations.
eIDI Summary Report [PDF, 710 KB]
- The evaluation report has more details about the project and is shorter and easier to read than the technical report.
eIDI Evaluation Report [PDF, 1.1 MB]
- A long form technical report of the e-IDI PoC, which includes data models, recommendations and appendices
e-IDI technical report [PDF, 1.8 MB]
Markdown versions of the two summary reports are included as appendices in the technical report from here http://purl.org/net/e-idi/doc/poc-technical-report
- The application for Better Public Services Seed Funding: Seed Fund Application
- The supporting documentation for the application, which includes the analysis and background documents: BPS seed fund application - supporting documentation
- A visual narrative of the application: A4 Visual Narrative
The website is established for the Environmental Integrated Data Infrastructure Proof of Concept (e-IDI POC) project and is for use during the e-IDI POC project. The information and data provided on this website are for the purposes of the e-IDI POC project only and are not recommended for operational use.