ARTÍCULO
TITULO

Scientific Big Data Visualization: a coupled tools approach

Antoni Artigues    
Fernando Martin Cucchietti    
Carlos Tripiana Montes    
David Vicente    
Hadrien Calmet    
Guillermo Marin    
Guillaume Houzeaux    
Mariano Vazquez    

Resumen

We designed and implemented a parallel visualisation system for the analysis of large scale time-dependent particle type data. The particular challenge we address is how to analyse a high perfor- mance computation style dataset when a visual representation of the full set is not possible or useful, and one is only interested in finding and inspecting smaller subsets that fulfil certain complex criteria. We used Paraview as the user interface, which is a familiar tool for many HPC users, runs in parallel, and can be conveniently extended. We distributed the data in a supercomputing environment using the Hadoop file system. On top of it, we run Hive or Impala, and implemented a connection between Paraview and them that al- lows us to launch programmable SQL queries in the database di- rectly from within Paraview. The queries return a Paraview-native VTK object that fits directly into the Paraview pipeline. We find good scalability and response times. In the typical supercomputer environment (like the one we used for implementation) the queue and management system make it difficult to keep local data in be- tween sessions, which imposes a bottleneck in the data loading stage. This makes our system most useful when permanently in- stalled on a dedicated cluster.

 Artículos similares

       
 
Abdelrahman Khalifa, Bashar Bashir, Abdullah Alsalman, Sambit Prasanajit Naik and Rosa Nappi    
Evaluating and predicting the occurrence and spatial remarks of climate and rainfall-related destructive hazards is a big challenge. Periodically, Sinai Peninsula is suffering from natural risks that enthuse researchers to provide the area more attention... ver más
Revista: Water

 
Dan Liu, Zhongkai Yao, Xiaoxia Yang, Chunmei Xiong and Qingyu Nie    
The agricultural non-point source (NPS) pollution caused by non-irrigated farming, such as heavy metals, nitrogen and phosphorus, has posed an extreme threat to the security of agricultural product quality and watershed ecology. Thus, it is urgent to sor... ver más
Revista: Water

 
Manoj Poudel, Rashmi P. Sarode, Yutaka Watanobe, Maxim Mozgovoy and Subhash Bhalla    
The rise of big data has resulted in the proliferation of numerous heterogeneous data stores. Even though multiple models are used for integrating these data, combining such huge amounts of data into a single model remains challenging. There is a need in... ver más
Revista: Applied Sciences

 
Lili Liang, Yufeng Hu, Zhiwu Liu, Yuntao Ye, Kuang Li, Kexin Liu, Haiqing Xu and Xiquan Liu    
The lumped hydrological model and empirical model have the problems of low accuracy and short forecasting period in real-time flood forecasting of small- and medium-sized rivers in a mountainous watershed. The sharing of underlying surface data such as h... ver más
Revista: Water

 
Jamil Al-Sawwa and Mohammad Almseidin    
With the rapid development of internet technology, the amount of collected or generated data has increased exponentially. The sheer volume, complexity, and unbalanced nature of this data pose a challenge to the scientific community to extract meaningful ... ver más
Revista: Information