Complex on-demand data retrieval and processing is a characteristic of several applications and combines the notions of querying & search, information filtering & retrieval, data transformation & analysis, and other data manipulations. Such rich tasks are typically represented by data processing graphs, having arbitrary data operators as nodes and their producer-consumer interactions as edges.
Interoperability is one of the most challenging issues in the design and operation of large-scale computing infrastructures for multidisciplinary research. It requires solutions that can “embrace” heterogeneity, i.e. accommodate multiple incarnations of similar resources, as well as “hide” it, i.e. provide consumers with a homogeneous view of diverse resources. This paper overviews the measures put in place in the D4Science infrastructure to address a range of interoperability problems.
Earth-Science data are composite, multidimensional and of significant size, and as such, continue to pose a number of on-going problems regarding their management. With new and diverse information sources emerging as well as rates of generated data continuously increasing, a persistent challenge becomes more pressing: to make the information existing in multiple heterogeneous resources readily available. The widespread use of the XML data-exchange format has enabled the rapid accumulation of semi-structured metadata for Earth-Science data.
Big Data Analytics is an emerging field since massive storage and computing capabilities have been made available by advanced e-infrastructures. Earth and Environmental sciences are likely to benefit from Big Data Analytics techniques supporting the processing of the large number of Earth Observation datasets currently acquired and generated through observations and simulations. However, Earth Science data and applications present specificities in terms of relevance of the geospatial information, wide heterogeneity of data models and formats, and complexity of processing.
The ever growing amount of information collected by scientific instruments and the presence of descriptive metadata accompanying them calls for a unified way of querying over array and semi-structured data. We present xWCPS, a novel query language that bridges the path between these two different worlds, enhancing the expressiveness and user-friendliness of previous approaches.
Catch a glimpse of our work and get inspired from the possibilities!
