Masters Thesis - Estimating Overlap of Verification and Testing Efforts Between Different Test Levels

Abstract

Context: Embedded systems (ES) are continuously growing in complexity and size in the automotive industry. The embedded software makes a larger part of the ES. Likewise, the testing process has become complex, larger, and difficult to control. This has led to the growing demand for test methods that would help to manage the testing process. Various methodologies are adapted in ES development. The structured layer testing "V-Model" is one of the highly used methodologies in ES testing. In V-Model, the testing is conducted in levels where each level focuses on its objectives, and the testing at one stage must be completed before entering another stage. A careful definition of test levels theoretically decreases the risk of gaps in testing between different teams and reduces the need for redundant testing efforts. Aim: This study proposes a framework to evaluate the existence and quantity of overlap and identify gaps between testing efforts at different levels of the Vdevelopment model during the development of embedded systems. Method: A case study was conducted at Scania CV AB, Södertälje. The firstdegree and third-degree data collection techniques were applied to gather insights about the testing process and systems in automotive embedded systems. Data collection and analysis were conducted in a parallel and iterative manner. Based on the insights gathered from data analyses, a mapping framework was designed to identify the overlap between test cases and gaps for the requirements not tested. The mapping framework was applied to three selected systems, and overlap and gaps were identified. In order to evaluate if identified overlap and gaps are valid, the results were verified by two testers from the case company. Results: The difference of V-model adaptation at the case company from the state of the art studied. Traceability of test cases, requirements, and functionalities are performed for implementing the framework. The proposed framework evaluated by applying on selected ECU (Electronic Control Unit) systems. The validation of framework evaluation result also performed by case company testers. The research addressed, quantitative measurements of overlaps and gaps between levels for different systems. Conclusion: The framework provided a promising result on identification of overlap and gaps between test levels automotive embedded system software. The study showed that, to identify the extent of overlap and gap between test levels, analysis of the test cases, requirements, and functionality can be one method. This study showed that, the proposed framework can be used for estimation of testing effort between different levels and for evaluating requirement traceability to the test case and functionality. The proposed framework can generally be applicable for automotive ES in the future by including more system samples.