Towards systematic trade-off management for MLOps: quality model, architectural tactics, design patterns

Abstract

Machine-learning-enabled systems are booming within modern software-intensive domains, driving innovation in areas such as healthcare, finance, and autonom ous systems. To produce high-quality and competitive solutions, the develop ment and operation of ML-enabled systems (MLOps) require careful considera tion of quality trade-offs across the entire production cycle. These trade-offs often differ from well-studied ones relevant to traditional software due to the inherently probabilistic and data-dependent nature of machine learning. Such differences particularly challenge startups and SMEs, who must operate in a non-standardized domain and seek frameworks that capture best practices for producing ML-enabled software. In this thesis, we propose a framework for embedding systematic trade off management into the MLOps lifecycle. Firstly, based on the results of a systematic literature review combined with an evaluation in industrial settings, we built a common quality model unique to ML-enabled systems. Second, through a systematic literature review and multiple-case study with four companies in the AI domain, we derived architectural and non-architectural tactics employed to achieve identified quality attributes. Third, we extracted existing design patterns from the component models of ML-enabled software identified through a multi-vocal literature review, and, using semi-structured expert interviews, evaluated their impact on quality attributes. In combination, these three studies lead to the insight that applying tactics or patterns to improve one quality attribute usually has side effects on other attributes, resulting in quality trade-offs. Our findings reveal that trade-off management is crucial for ML software production, as it significantly influences decision-making at all dimensions of the MLOps lifecycle (data, model, operations, and development). Based on these insights, as a fourth and final contribution, we propose a vision of an extension to the overall MLOps paradigm by embedding explicit steps for trade-off management at all phases of the workflow.Machine-learning-enabled systems are booming within modern software-intensive domains, driving innovation in areas such as healthcare, finance, and autonom ous systems. To produce high-quality and competitive solutions, the develop ment and operation of ML-enabled systems (MLOps) require careful considera tion of quality trade-offs across the entire production cycle. These trade-offs often differ from well-studied ones relevant to traditional software due to the inherently probabilistic and data-dependent nature of machine learning. Such differences particularly challenge startups and SMEs, who must operate in a non-standardized domain and seek frameworks that capture best practices for producing ML-enabled software. In this thesis, we propose a framework for embedding systematic trade off management into the MLOps lifecycle. Firstly, based on the results of a systematic literature review combined with an evaluation in industrial settings, we built a common quality model unique to ML-enabled systems. Second, through a systematic literature review and multiple-case study with four companies in the AI domain, we derived architectural and non-architectural tactics employed to achieve identified quality attributes. Third, we extracted existing design patterns from the component models of ML-enabled software identified through a multi-vocal literature review, and, using semi-structured expert interviews, evaluated their impact on quality attributes. In combination, these three studies lead to the insight that applying tactics or patterns to improve one quality attribute usually has side effects on other attributes, resulting in quality trade-offs. Our findings reveal that trade-off management is crucial for ML software production, as it significantly influences decision-making at all dimensions of the MLOps lifecycle (data, model, operations, and development). Based on these insights, as a fourth and final contribution, we propose a vision of an extension to the overall MLOps paradigm by embedding explicit steps for trade-off management at all phases of the workflow.