Performance Models for Distributed-memory HPC Systems and Deep Neural Networks

David Cardwell, Purdue University

Abstract

Performance models are useful as mathematical models to reason about the behavior of different computer systems while running various applications. An additional purpose is predicting performance without having to run a given application on a target system. Models that simultaneously provide accurate results as well as insight in optimization techniques are difficult to develop since more accurate models tend to be more complex. Straightforward and precise models are of great interest for this reason. In this thesis, we aim to provide two distinct performance models: one for distributedmemory high performance computing systems with network communication, and one for deep neural networks. Our main goal for the first model is insight and simplicity, while for the second we aim for accuracy in prediction. The first model is generalized for networked multi-core computer systems, while the second is specific to deep neural networks on a shared-memory system. First, we enhance the well-known Roofline model with extensions to add communication awareness. To do so, we introduce the concept of communication arithmetic intensity, which is the network equivalent of operational intensity. In the second model, we use performance measurements from target systems to parameterize the model by the problem size. For both models, we performed an empirical analysis on several target systems with various algorithms and applications to verify the quality of the predictions. With our communication-aware extended Roofline model, we improve on the original model by up to 100% on all three tested computer systems according to our MAPE-based comparision method. For the deep neural network model, we attain up to a 23.43% absolute prediction error. To our knowledge, our communication-aware Roofline model is the first Roofline extension to consider communication, while our second model is the first model of deep neural networks that uses parameterization.

Degree

M.Sc.

Advisors

Song, Purdue University.

Subject Area

Communication|Artificial intelligence|Computer science|Mathematics

COinS