Supported by the National Science Foundation Collaborator: University of Michigan Collaborator: Michigan State University Collaborator: Wayne State University Collaborator: Indiana University

Moving USGS Data With DLT And OSiRIS At SC16 Rapid Distribution of Earth Science Data with the Earth Observation Depot Network and the Data Logistics Toolkit

16 November 2016
DLT Data Movement Visualization

The NSF-funded Data Logistics Toolkit (DLT) project was featured in the Indiana University and the University of Michigan Advanced Research Computing booths at Supercomputing in Salt Lake City this year.

As an instantiation of the DLT, the Earth Observation Depot Network (EODN) aims to enable open access, reduced latency, and fast downloads for valuable and compelling Earth Science data from satellites for meteorological and atmospheric researchers. Data sources include remote sensing data from NASA’s Earth Science Program (EOS-DIS) and the United States Geological Survey (USGS) Landsat ground network and are made available within EODN via a “harvester” workflow maintained at IU.

DLT Data Movement Visualization Data from NASA’s Earth Science Program (EOS-DIS) and the United States Geological Survey (USGS) Landsat ground network transferring from OSiRIS and Cloudlab using DLT

Initially deployed on GENI infrastructure, EODN has been extended to operate on the OSiRIS open storage infrastructure using Ceph technology. In addition, we made use of resources at Utah’s CloudLab to show the viability of a cloud-based storage cluster that may be instantiated during periods of high demand. Our demonstration at Supercomputing highlighted this interoperability and evaluated the performance of large-scale Landsat data transfers between Michigan, Indiana, and Salt Lake City storage sites.

SC16 network weather map Weathermap of DLT network data transfers between OSiRIS sites at Michigan, SC16, and separate cluster at Cloudlab.

The Supercomputing network infrastructure provided a valuable testbed to further research and development on a number of DLT and OSiRIS components:

  • libdlt – DLT filesystem middleware. Includes an object store abstraction for the interchangeable use of underlying data movement routines.

  • IDMS – The Intelligent Data Movement Service enforces global and user-specific data distribution policies and governs the storage nodes between which data is transferred.

  • UNIS-RT and Flange – The runtime for the Unified Network Information Service (UNIS). Flange directly uses the runtime to operate programmatically on a UNIS topology representation (network graph) for dynamic updates and control.

  • Periscope – Measurement collection/analysis framework.

More information on DLT and EODN may be found at data-logistics.org

Poster and White paper from the Rapid Earth Science demo at SC16.

Thanks to Ezra Kissel, Jayashree Ajay Candadai, Jeremy Musser, and the rest of the OSiRIS team at IU for orchestrating this demo and capturing visualizations.

Tags