Techniques for Enhancing the Efficiency and Trustworthiness of Neural Networks

<p dir="ltr">This dissertation addresses the problem of enhancing the efficiency and trustworthiness of Neural Network (NN) models. </p><p dir="ltr">On the efficiency side, my work focuses on the problem of Hyper-parameter tuning (HPT), a crucial but expensive step to optimize the performance of NN. In this area, my dissertation introduces two novel HPT methods, HyperJump (HJ) [119] and TrimTuner [118]. Both methods aim to maximize model’s quality while reducing training and optimization time. Despite using different techniques to solve the optimization problem, both rely on low-fidelity observations (e.g., training with sub sampled datasets) to efficiently identify promising configurations to be then tested via high-fidelity observations (e.g., using the full dataset). </p><p dir="ltr">On the trustworthiness side, the focus of this work is on adversarial robustness and uncertainty estimation. On the adversarial robustness front, I investigate the challenges that arise when performing HPT for models that are adversarially trained, showing that, although the complexity of the HPT problem is exacerbated in adversarial settings, tuning the HPs independently for standard and Adversarial Training (AT) can improve accuracy. To reduce HPT costs, I propose leveraging weaker, but cheaper, AT methods to obtain inexpensive, yet highly correlated, estimations of the quality of more robust methods. This approach, combined with a recent multi fidelity optimizer, enhances the efficiency of the HPT process by up to 2.1×. </p><p dir="ltr">On the uncertainty estimation front, I first introduce Error-Driven Uncertainty Aware Training (EUAT) [120], a method that strives to ensure that the model is highly uncertain when making inaccurate predictions and confident when making accurate ones. During training EUAT selectively employs two loss functions that operate at training time with the twofold goal of: i) reducing uncertainty for correct predictions and ii) increasing uncertainty for mispredictions, while preserving the accuracy. However, EUAT is applicable exclusively to classification tasks and entails a non-negligible computational overhead. These limitations are addressed by the last solution introduced with my dissertation, namely CLUE (Calibration via Learning Uncertainty–Error Alignment). This method introduces a general calibra tion framework for NNs that aligns predictive uncertainty with model error through a scalable and differentiable loss. Together, EUAT and CLUE advance the broader goal of enhancing model trustworthiness by generating uncertainty estimates that are well-calibrated. </p><p dir="ltr">Overall, this dissertation contributes to the growing body of research aimed at making Machine Learning systems not only more accurate, but also more efficient and trustworthy by incorporating fidelity-aware optimization, robust adversarial training, and calibrated uncertainty estimation into the training process.</p>