Scaling big data platform for big data pipeline
February 11, 2019
Abstract
Author:
Published in:
Submitted to Northeast Database Day, NEBD 2020, https://arxiv.org/abs/1902.03948
R&D Area:
R&D Group:
Summary
Monitoring and Managing High Performance Computing (HPC) systems and environments generate an ever growing amount of data. Making sense of this data and generating a platform where the data can be visualized for system administrators and management to proactively identify system failures or understand the state of the system requires the platform to be as efficient and scalable as the underlying database tools used to store and analyze the data. In this paper we will show how we leverage Accumulo, d4m, and Unity to generate a 3D visualization platform to monitor and manage the Lincoln Laboratory Supercomputer systems and how we have had to retool our approach to scale with our systems.