Carnegie Mellon University
Browse

Self-Adaptive Machine Learning-based System

Download (5.94 MB)
thesis
posted on 2025-03-18, 21:14 authored by Maria Da Loura CasimiroMaria Da Loura Casimiro

Machine learning (ML) models are now commonly used as components in systems. Due to their black-box and data-driven nature, ML components can produce erroneous outputs (in the form of mispredictions) that may critically impact the system’s quality of service. Such mispredictions may be caused by component changes, environment changes, or due to ML components’ inherent uncertainty and inaccuracy. In the face of these changes, and to cope with mis predictions, self-adaptation arises as a natural solution: systems that monitor and adapt themselves at run time to optimize their system utility.

This thesis provides a repertoire of ML adaptation tactics, and a framework that generates policies specifying when to apply each tactic to adapt ML com ponents such that overall system utility is optimized. The development of this framework raises two main challenges: (i) estimating the expected costs and benefits due to the execution of an adaptation tactic; (ii) evaluating the impact of the improved ML predictions on overall system utility. To address the first problem we build predictors that learn to estimate the expected benefits of each adaptation tactic. To solve the second problem we leverage probabilistic model checking methods and instantiate a formal model of the system, capturing the key dynamics of ML components and their impact on expected system utility.

The techniques proposed in this thesis are evaluated via two use-cases: credit card fraud detection and machine translation systems. We show that: the self-adaptive ML-based systems built leveraging the proposed framework achieve better system utility than that achievable when employing simpler base lines to guide adaptation, such as periodic or reactive adaptation schemes; the proposed framework is suitable for run-time adaptation in non-critical domains; the framework can be extended to account for multiple adaptation tactics; and that it can be leveraged to plan for the long term when to adapt ML models.

Funding

CMU-PT Program, Portuguese National Science Foundation (Fundacao para a Ciencia e Tecnologia (FCT))

History

Date

2024-12-19

Degree Type

  • Dissertation

Department

  • Software and Societal Systems (S3D)

Degree Name

  • Doctor of Philosophy (PhD)

Advisor(s)

David Garlan (CMU) Paolo Romano (IST)

Usage metrics

    Exports

    RefWorks
    BibTeX
    Ref. manager
    Endnote
    DataCite
    NLM
    DC