Enhancing Performance Evaluation and Characterization Techniques to I dentify Performance Changes in High-Performance Computing Systems

Abstract

High Performance Computing systems are complex and require a lot of effort to tune the system to achieve peak performance. Performance analysis is a time consuming process. The goal of this thesis is to understand the effects changes to the system or compiler configuration had on performance and how it is reflected in CPU performance metrics. This thesis presents two approaches to enhance the evaluation process of HPC systems. First, a process that makes it possible to systematically and efficiently search the parameter space to find an optimal configuration of a benchmark with a large number of tunable parameters is introduced. The search for an optimal combination of parameters can be daunting, especially when it involves high dimensionality of mixed type categorical and continuous variables. This thesis shows that through the use of statistical techniques, a systematic and efficient search of the parameter space can be conducted. These techniques can be applied to variables that are categorical or continuous in nature and do not rely on the standard assumptions of linear models, namely that the response variable can be described as a linear combination of the regression coefficients. Second, a normalization technique that will make it possible to identify relative differences between performance metrics to better understand the effects changes had on the underlying system is presented. The use of Top-Down microarchitecture analysis method makes it possible to understand how pipeline bottlenecks were affected by changes in the system configuration, or compiler version. Bottleneck analysis makes it possible to better understand how different hardware resources are being utilized, highlighting portions of the CPU's pipeline where possible improvements could be achieved. The Top-Down analysis method is complemented with the use a normalization technique from the field of economics, purchasing power parity (PPP), to better understand the relative difference between changes. This thesis showed that system changes had effects that sometimes were not reflected on the corresponding Top-Down metrics. The use of the PPP normalization technique made it possible to highlight differences and trends in bottleneck metrics, differences that standard techniques based in absolute, non-normalized, metrics failed to highlight.