Contributions to Optimal Experimental Designs with Application to the Development of Machine Learning Based Systems

Abstract

Design of experiments (DoE) is long-established as an indispensable methodology for reliableand expeditious product development across many domains. Over the years, the application context of DoE has evolved from agriculture experiments to industrial experiments, and more recently, to experiments on software products. The three studies in the thesis make methodological contributions to the optimal experimental design literature and propose experimentation strategies for the following objectives (i) for optimizing an unknown noisy black-box function, (ii) for estimating an unknown noisy black-box function, and (iii) for identifying the best surrogate model among m ≥ 2 candidate models that best approximate the unknown black-box function. The application context of the first two studies is hyperparameter optimization of machine learning models, a critical step in their training process. The application context of the third study is online evaluation of machine learning models, a critical step for validating the performance of the models prior to their deployment. We illustrate the utility of the proposed experimentation strategies through simulation studies on synthetic test functions and two case studies at a large medical device manufacturer in the context of automating visual quality inspections in manufacturing. Minimizing the total cost of experimentation and shortening the experimentation lifecycle for the development of reliable machine learning based systems are the key contributions of the methods proposed in this thesis.