Publications

Refine Results

(Filters Applied) Clear All

Tagged As

Streaming 1.9 billion hyperspace network updates per second with D4M

September 24, 2019
  |
Conference Paper
Author:
Jeremy Kepner
…
Published in:
2019 IEEE High Performance Computing Conf., HPEC, 24-26 September 2019.
Topic:
high performance computing
R&D area:
R&D group:

Summary

The Dynamic Distributed Dimensional Data Model (D4M) library implements associative arrays in a variety of languages (Python, Julia, and Matlab/Octave) and provides a lightweight in-memory database implementation of hypersparse arrays that are ideal for analyzing many types of network data. D4M relies on associative arrays which combine properties of spreadsheets, databases, matrices, graphs, and networks, while providing rigorous mathematical guarantees, such as linearity. Streaming updates of D4M associative arrays put enormous pressure on the memory hierarchy. This work describes the design and performance optimization of an implementation of hierarchical associative arrays that reduces memory pressure and dramatically increases the update rate into an associative array. The parameters of hierarchical associative arrays rely on controlling the number of entries in each level in the hierarchy before an update is cascaded. The parameters are easily tunable to achieve optimal performance for a variety of applications. Hierarchical arrays achieve over 40,000 updates per second in a single instance. Scaling to 34,000 instances of hierarchical D4M associative arrays on 1,100 server nodes on the MIT SuperCloud achieved a sustained update rate of 1,900,000,000 updates per second. This capability allows the MIT SuperCloud to analyze extremely large streaming network data sets.
READ LESS

Summary

The Dynamic Distributed Dimensional Data Model (D4M) library implements associative arrays in a variety of languages (Python, Julia, and Matlab/Octave) and provides a lightweight in-memory database implementation of hypersparse arrays that are ideal for analyzing many types of network data. D4M relies on associative arrays which combine properties of spreadsheets...

READ MORE
Streaming 1.9 billion hyperspace network updates per second with D4M

Scaling big data platform for big data pipeline

February 11, 2019
  |
Abstract
Author:
Rebecca L. Wild
…
Published in:
Submitted to Northeast Database Day, NEBD 2020, https://arxiv.org/abs/1902.03948
Topic:
high performance computing
R&D area:
R&D group:

Summary

Monitoring and Managing High Performance Computing (HPC) systems and environments generate an ever growing amount of data. Making sense of this data and generating a platform where the data can be visualized for system administrators and management to proactively identify system failures or understand the state of the system requires the platform to be as efficient and scalable as the underlying database tools used to store and analyze the data. In this paper we will show how we leverage Accumulo, d4m, and Unity to generate a 3D visualization platform to monitor and manage the Lincoln Laboratory Supercomputer systems and how we have had to retool our approach to scale with our systems.
READ LESS

Summary

Monitoring and Managing High Performance Computing (HPC) systems and environments generate an ever growing amount of data. Making sense of this data and generating a platform where the data can be visualized for system administrators and management to proactively identify system failures or understand the state of the system requires...

READ MORE
Scaling big data platform for big data pipeline

A billion updates per second using 30,000 hierarchical in-memory D4M databases

January 24, 2019
  |
Conference Paper
Author:
Jeremy Kepner
…
Published in:
North East Database Day, NEDB Day, 24 January 2019.
Topic:
high performance computing
R&D area:
R&D group:

Summary

Analyzing large scale networks requires high performance streaming updates of graph representations of these data. Associative arrays are mathematical objects combining properties of spreadsheets, databases, matrices, and graphs, and are well-suited for representing and analyzing streaming network data. The Dynamic Distributed Dimensional Data Model (D4M) library implements associative arrays in a variety of languages (Python, Julia, and Matlab/Octave) and provides a lightweight in-memory database. Associative arrays are designed for block updates. Streaming updates to a large associative array requires a hierarchical implementation to optimize the performance of the memory hierarchy. Running 34,000 instances of a hierarchical D4M associative arrays on 1,100 server nodes on the MIT SuperCloud achieved a sustained update rate of 1,900,000,000 updates per second. This capability allows the MIT SuperCloud to analyze extremely large streaming network data sets.
READ LESS

Summary

Analyzing large scale networks requires high performance streaming updates of graph representations of these data. Associative arrays are mathematical objects combining properties of spreadsheets, databases, matrices, and graphs, and are well-suited for representing and analyzing streaming network data. The Dynamic Distributed Dimensional Data Model (D4M) library implements associative arrays in...

READ MORE
A billion updates per second using 30,000 hierarchical in-memory D4M databases

Hyperscaling internet graph analysis with D4M on the MIT SuperCloud

September 26, 2018
  |
Conference Paper
Author:
Vijay N. Gadepally
…
Published in:
IEEE High Performance Extreme Computing Conf., HPEC, 25-27 September 2018.
Topic:
high performance computing
R&D area:
R&D group:

Summary

Detecting anomalous behavior in network traffic is a major challenge due to the volume and velocity of network traffic. For example, a 10 Gigabit Ethernet connection can generate over 50 MB/s of packet headers. For global network providers, this challenge can be amplified by many orders of magnitude. Development of novel computer network traffic analytics requires: high level programming environments, massive amount of packet capture (PCAP) data, and diverse data products for "at scale" algorithm pipeline development. D4M (Dynamic Distributed Dimensional Data Model) combines the power of sparse linear algebra, associative arrays, parallel processing, and distributed databases (such as SciDB and Apache Accumulo) to provide a scalable data and computation system that addresses the big data problems associated with network analytics development. Combining D4M with the MIT SuperCloud manycore processors and parallel storage system enables network analysts to interactively process massive amounts of data in minutes. To demonstrate these capabilities, we have implemented a representative analytics pipeline in D4M and benchmarked it on 96 hours of Gigabit PCAP data with MIT SuperCloud. The entire pipeline from uncompressing the raw files to database ingest was implemented in 135 lines of D4M code and achieved speedups of over 20,000.
READ LESS

Summary

Detecting anomalous behavior in network traffic is a major challenge due to the volume and velocity of network traffic. For example, a 10 Gigabit Ethernet connection can generate over 50 MB/s of packet headers. For global network providers, this challenge can be amplified by many orders of magnitude. Development of...

READ MORE
Hyperscaling internet graph analysis with D4M on the MIT SuperCloud

Simulation approach to sensor placement using Unity3D

September 25, 2018
  |
Conference Paper
Author:
Kimberlee Chestnut Chang
…
Published in:
IEEE High Performance Extreme Computing Conf., HPEC, 25-27 September 2018.
Topic:
high performance computing
R&D area:
R&D group:

Summary

3D game simulation engines have demonstrated utility in the areas of training, scientific analysis, and knowledge solicitation. This paper will make the case for the use of 3D game simulation engines in the field of sensor placement optimization. Our study used a series of parallel simulations in the Unity3D simulation framework to answer the questions: how many sensors of various modalities are required and where they should be placed to meet a desired threat detection threshold? The result is a framework that not only answers this sensor placement question, but can be easily expanded to differing optimization criteria as well as answer how a particular configuration responds to differing crowd flows or informed/non-informed adversaries. Additionally, we demonstrate the scalability of this framework by running parallel instances on a supercomputing grid and illustrate the processing speed gained.
READ LESS

Summary

3D game simulation engines have demonstrated utility in the areas of training, scientific analysis, and knowledge solicitation. This paper will make the case for the use of 3D game simulation engines in the field of sensor placement optimization. Our study used a series of parallel simulations in the Unity3D simulation...

READ MORE
Simulation approach to sensor placement using Unity3D

A parallel implementation of FANO using OpenMP and MPI

September 25, 2018
  |
Conference Paper
Author:
Plamen Krastev
…
Published in:
IEEE High Performance Extreme Computing Conf., HPEC, 25-27 September 2018.
Topic:
high performance computing
R&D area:
R&D group:

Summary

We present a parallel implementation of the Fast Accurate NURBS Optimization (FANO) program using OpenMP and MPI. The software is used for designing imaging freeform optical systems comprised of NURBS surfaces. An important step in the design process is the optimization of the shape and position of the optical surfaces within the optical system. FANO uses the Levenberg-Marquardt (LM) algorithm for minimization of the merit function. The parallelization of the code is achieved without modifying readily available commercial or open source implementations of the LM algorithm. Instead, MPI instructions are being used to distribute the computation of the Jacobian over multiple nodes, each of which performs the computationally intensive task of raytracing. The results from the raytracing are collected on the master and used for calculating the values of the variable parameters for the next iteration. Speed increases of ~100x and more are possible when running on the cluster of the MIT Lincoln Laboratory Super Computing Center (LLSC).
READ LESS

Summary

We present a parallel implementation of the Fast Accurate NURBS Optimization (FANO) program using OpenMP and MPI. The software is used for designing imaging freeform optical systems comprised of NURBS surfaces. An important step in the design process is the optimization of the shape and position of the optical surfaces...

READ MORE
A parallel implementation of FANO using OpenMP and MPI

GraphChallenge.org: raising the bar on graph analytic performance

September 25, 2018
  |
Conference Paper
Author:
Siddharth S. Samsi
…
Published in:
IEEE High Performance Extreme Computing Conf., HPEC, 25-27 September 2018.
Topic:
high performance computing
R&D area:
R&D group:

Summary

The rise of graph analytic systems has created a need for new ways to measure and compare the capabilities of graph processing systems. The MIT/Amazon/IEEE Graph Challenge has been developed to provide a well-defined community venue for stimulating research and highlighting innovations in graph analysis software, hardware, algorithms, and systems. GraphChallenge.org provides a wide range of preparsed graph data sets, graph generators, mathematically defined graph algorithms, example serial implementations in a variety of languages, and specific metrics for measuring performance. Graph Challenge 2017 received 22 submissions by 111 authors from 36 organizations. The submissions highlighted graph analytic innovations in hardware, software, algorithms, systems, and visualization. These submissions produced many comparable performance measurements that can be used for assessing the current state of the art of the field. There were numerous submissions that implemented the triangle counting challenge and resulted in over 350 distinct measurements. Analysis of these submissions show that their execution time is a strong function of the number of edges in the graph, Ne, and is typically proportional to N4=3 e for large values of Ne. Combining the model fits of the submissions presents a picture of the current state of the art of graph analysis, which is typically 108 edges processed per second for graphs with 108 edges. These results are 30 times faster than serial implementations commonly used by many graph analysts and underscore the importance of making these performance benefits available to the broader community. Graph Challenge provides a clear picture of current graph analysis systems and underscores the need for new innovations to achieve high performance on very large graphs.
READ LESS

Summary

The rise of graph analytic systems has created a need for new ways to measure and compare the capabilities of graph processing systems. The MIT/Amazon/IEEE Graph Challenge has been developed to provide a well-defined community venue for stimulating research and highlighting innovations in graph analysis software, hardware, algorithms, and systems...

READ MORE
GraphChallenge.org: raising the bar on graph analytic performance

TabulaROSA: tabular operating system architecture for massively parallel heterogeneous compute engines

September 25, 2018
  |
Conference Paper
Author:
Jeremy Kepner
…
Published in:
IEEE High Performance Extreme Computing Conf., HPEC, 25-27 September 2018.
Topic:
high performance computing
R&D area:
R&D group:

Summary

The rise in computing hardware choices is driving a reevaluation of operating systems. The traditional role of an operating system controlling the execution of its own hardware is evolving toward a model whereby the controlling processor is distinct from the compute engines that are performing most of the computations. In this context, an operating system can be viewed as software that brokers and tracks the resources of the compute engines and is akin to a database management system. To explore the idea of using a database in an operating system role, this work defines key operating system functions in terms of rigorous mathematical semantics (associative array algebra) that are directly translatable into database operations. These operations possess a number of mathematical properties that are ideal for parallel operating systems by guaranteeing correctness over a wide range of parallel operations. The resulting operating system equations provide a mathematical specification for a Tabular Operating System Architecture (TabulaROSA) that can be implemented on any platform. Simulations of forking in TabularROSA are performed using an associative array implementation and compared to Linux on a 32,000+ core supercomputer. Using over 262,000 forkers managing over 68,000,000,000 processes, the simulations show that TabulaROSA has the potential to perform operating system functions on a massively parallel scale. The TabulaROSA simulations show 20x higher performance as compared to Linux while managing 2000x more processes in fully searchable tables.
READ LESS

Summary

The rise in computing hardware choices is driving a reevaluation of operating systems. The traditional role of an operating system controlling the execution of its own hardware is evolving toward a model whereby the controlling processor is distinct from the compute engines that are performing most of the computations. In...

READ MORE
TabulaROSA: tabular operating system architecture for massively parallel heterogeneous compute engines

Measuring the impact of Spectre and Meltdown

September 25, 2018
  |
Conference Paper
Author:
Andrew J. Prout
…
Published in:
IEEE High Performance Extreme Computing Conf., HPEC, 25-27 September 2018.
Topic:
high performance computing
R&D area:
R&D group:

Summary

The Spectre and Meltdown flaws in modern microprocessors represent a new class of attacks that have been difficult to mitigate. The mitigations that have been proposed have known performance impacts. The reported magnitude of these impacts varies depending on the industry sector and expected workload characteristics. In this paper, we measure the performance impact on several workloads relevant to HPC systems. We show that the impact can be significant on both synthetic and realistic workloads. We also show that the performance penalties are difficult to avoid even in dedicated systems where security is a lesser concern.
READ LESS

Summary

The Spectre and Meltdown flaws in modern microprocessors represent a new class of attacks that have been difficult to mitigate. The mitigations that have been proposed have known performance impacts. The reported magnitude of these impacts varies depending on the industry sector and expected workload characteristics. In this paper, we...

READ MORE
Measuring the impact of Spectre and Meltdown

Interactive supercomputing on 40,000 cores for machine learning and data analysis

September 25, 2018
  |
Conference Paper
Author:
Albert I. Reuther
…
Published in:
IEEE High Performance Extreme Computing Conf., HPEC, 25-27 September 2018.
Topic:
high performance computing
R&D area:
R&D group:

Summary

Interactive massively parallel computations are critical for machine learning and data analysis. These computations are a staple of the MIT Lincoln Laboratory Supercomputing Center (LLSC) and has required the LLSC to develop unique interactive supercomputing capabilities. Scaling interactive machine learning frameworks, such as TensorFlow, and data analysis environments, such as MATLAB/Octave, to tens of thousands of cores presents many technical challenges – in particular, rapidly dispatching many tasks through a scheduler, such as Slurm, and starting many instances of applications with thousands of dependencies. Careful tuning of launches and prepositioning of applications overcome these challenges and allow the launching of thousands of tasks in seconds on a 40,000-core supercomputer. Specifically, this work demonstrates launching 32,000 TensorFlow processes in 4 seconds and launching 262,000 Octave processes in 40 seconds. These capabilities allow researchers to rapidly explore novel machine learning architecture and data analysis algorithms.
READ LESS

Summary

Interactive massively parallel computations are critical for machine learning and data analysis. These computations are a staple of the MIT Lincoln Laboratory Supercomputing Center (LLSC) and has required the LLSC to develop unique interactive supercomputing capabilities. Scaling interactive machine learning frameworks, such as TensorFlow, and data analysis environments, such as...

READ MORE
Interactive supercomputing on 40,000 cores for machine learning and data analysis
11-20 of 69