Defining and Validating a Multimodel Approach for Product Architecture Derivation and Improvement

Javier Gonzlez-Huerta, Emilio Insfran, Silvia Abraho

ISSI Research Group, Universitat Politcnica de Valncia Camino de Vera, s/n, 46022, Valencia, Spain
{jagonzalez, einsfran, sabrahao}@dsic.upv.es

Outline

Introduction
Software Product Line Product Line Architecture vs Product Architecture

QuaDAI: Architecture Derivation and Improvement Method

Validation Through a Controlled Experiment
Experiment planning Experiment operation and execution

Data analysis
Threats to the validity

Conclusions

Introduction

Software Product Line (SPL) development is an approach that takes advantage of the massive reuse of software assets to improve productivity and product quality In SPL development, there are two architectures:
the product line architecture contains a set of variation mechanisms that support the quality and functional requirements of the entire set of products that constitute the product line the product architecture is derived from the product line architecture by exercising its built-in variation mechanisms

Once the product architecture is derived, it should be evaluated to guarantee that meets the product specific NFRs
and.. when these NFRs cannot be attained by using built-in variation mechanisms certain architectural transformations may be applied

Objective

To introduce a model-driven product architecture derivation and improvement method (QuaDAI) for SPL aimed to: derive product architectures from the product line architecture

evaluate the product architecture obtained

when required, to improve the quality of the product architecture by applying pattern-based architectural transformations

To empirically validate the method using a controlled experiment

it evaluates the effectiveness, efficiency, perceived ease of use, usefulness and intention to use of the method in comparison with the Architecture Trade-Off Analysis Method (ATAM)

The QuaDAI method

QuaDAI is a model-driven method for the derivation, evaluation and improvement of product architectures which is composed of:
a multimodel to represent different viewpoints (functionality, quality, variability, etc.) as interrelated models a process consisting of a set of activities conducted by model transformations

There are other approaches for architecture derivation and evaluation:

SAAM does not consider interactions amongst competing quality attributes (not for SPL) ATAM provides a principled way to evaluate the fitness of a software architecture with respect to multiple competing quality attributes (not for SPL) EATAM and HoPLAA (extensions of ATAM for SPL) to assess the quality of both product line and product architectures but lack a systematic mechanism for architectural improvement

The QuaDAI multimodel 1/1

A multimodel is a set of models that represent different viewpoints of a system
A viewpoint* produces a specification of the system restricted to a particular concern (e.g., functionality, quality, variability) A viewpoint of interest is the Transformation Viewpoint, which contains the representation of architectural patterns which can be, for example, applied to improve a given quality attribute

The multimodel also contains relationships among the elements of different viewpoints This can be used to establish relationships of interest, for example:
how a specific component impact on the reliability of the system how a specific transformation improves a given quality attribute

* In a viewpoint, it is possible to define model of the system that contains only the objects that are visible from that viewpoint. Such a model is known as a viewpoint model of the system from that viewpoint (NISTIR 6928, 2003)
6

The QuaDAI multimodel 2/2

A specific instance of a multimodel:

The QuaDAI process

Includes different activities in which the multimodel is used to drive the model transformation processes for the derivation, evaluation and improvement of product architectures in SPL development

Activity 1

Activity 2

Activity 3

Activity 1. Product Architecture Derivation

Application Engineer PL Architecture Product Architecture Derivation

out

Evaluator

Architect

Configuration (Product Features + NFRs)

Product Architecture Evaluation

Product Architecture Transformation

Multimodel * Variability View * Functional View * Transformation View * Quality View

Product Architecture

Activity 2. Product Architecture Evaluation

Application Engineer PL Architecture Product Architecture Derivation

Evaluator

Architect

Configuration (Product Features + NFRs)

Product Architecture Evaluation

out

Product Architecture Transformation

Multimodel * Variability View * Functional View * Transformation View * Quality View

Product Architecture

Evaluation Report

10

Activity 3. Product Architecture Transformation

Application Engineer PL Architecture Product Architecture Derivation

Evaluator

Architect

Configuration (Product Features + NFRs)

Product Architecture Evaluation

Product Architecture Transformation

out

Multimodel * Variability View * Functional View * Transformation View * Quality View

Product Architecture

Product Architecture (improved)

11

Example. Activity 3. Product Architecture Transformation (ABS System)

A software product architecture of an automotive control system The system is composed of several software components that
Capture input signals from sensors Process and transform these inputs Control the actuators of different components of a car

The product architecture for an Antilock Braking System (ABS) system:

12

Example. Activity 3. Product Architecture Transformation (ABS System)

Excerpt of a multimodel including the quality and the transformation view:

Having a NFR regarding fault tolerance which architectural transformation pattern will be applied?

relation TripleModularRedundancy { checkonly domain Source: ... {...}; enforce domain Target: ... {...}; } relation Watchdog { checkonly domain Source: ... {...}; enforce domain Target: ... {...}; }

13

Example. Activity 3. Product Architecture Transformation (ABS System) Obtaning a Fault Tolerant software architecture
Having a NFR regarding fault tolerance the transformation pattern automatically applied will be the TMR pattern

14

Outline

Introduction
Software Product Line Product Line Architecture vs Product Architecture

QuaDAI: Architecture Derivation and Improvement Method

Validation through a controlled experiment
Experiment planning Experiment operation and execution

Data analysis
Threats to the validity

Conclusions

15

Validation: Experiment planning

The goal of the experiment was to
analyze QuaDAI and ATAM for the purpose of evaluating them with regard to their effectiveness, efficiency, and perceived utility (measured by means of 3 subjective variables: ease of use, usefulness and intention to use) in order to obtain software architectures that meet a given set of quality requirements from the viewpoint of novice software architecture evaluators

We focus on 2nd and 3rd phases of QuaDAI:

Product Architecture Evaluation Product Transformation These two activities deal with the evaluation and improvement of product architectures, which are aligned with the main purpose of ATAM (derivation is not included in ATAM)

16

Validation: Experiment planning. Context 1/2

The software architectures to be evaluated were: The software architecture of an Antilock Braking System (ABS System) from an automobile control system

The software architecture of a mobile application for emergency notifications

4 architectural patterns to be applied to improve the quality attribute levels in each of the product architectures

The experimental tasks include the evaluation of these quality attributes by means of two software metrics in each experimental object before and after applying the architecture evaluation methods (QuaDAI and ATAM)

17

Validation: Experiment planning. Context 2/2

Architecture Evaluation Methods
QuaDAI has been compared with the ATAM
ATAM is a widely used software architecture evaluation method ATAM is able to deal with multi-attribute analysis ATAM can be used to evaluate both product line and product architectures at various stages of SPL development (conceptual, before code, during development, or after deployment)

Subjects Selection
31 subjects were selected from a group of fifth-year Computer Science students at the Universitat Politcnica de Valncia who were enrolled on an Advanced Software Engineering course from September 2012 to January 2013

18

Validation: Experiment planning. Variables

The independent variable was the use of each method (ATAM or QuaDAI). The objective dependent variables:
Effectiveness of the method, calculated as a function of the Euclidean Distances between the NFR value attained by the subject and the optimal NFR value that can be attained Efficiency, calculated as the ratio between the effectiveness and the total time spent on the evaluation method

The Perceived Utility which has been measured by means of 3 subjective dependent variables:
Perceived ease of use (PEOU), the degree to which evaluators believe that learning and using a particular method will be effort-free Perceived usefulness (PU), the degree to which evaluators believe that using a specific method will increase their job performance Intention to Use (ITU), which is the extent to which a person intends to use a particular method

19

Validation: Experiment planning. Hypotheses

The hypotheses of the experiment were: H10: There is no significant difference between the effectiveness of QuaDAI and ATAM / H1a: QuaDAI is significantly more effective than ATAM H20: There is no significant difference between the efficiency of QuaDAI and ATAM / H2a: QuaDAI is significantly more efficient than ATAM. H30: There is no significant difference between the perceived ease of use of QuaDAI and ATAM / H3a: QuaDAI is perceived as easier to use than ATAM.

H40: There is no significant difference between the perceived usefulness of QuaDAI and ATAM / H4a: QuaDAI is perceived as more useful than ATAM.
H50: There is no significant difference between the perceived intention to use of QuaDAI and ATAM / H5a: QuaDAI is perceived as more likely to be used than ATAM.

20

Validation: Experiment operation and execution

The experiment was planned as a balanced within-subject design with a confounding effect
the same subjects executed both methods with both experimental objects in different order

1st session (120 min) session (60 + 60 minutes) 2nd

Software Architecture Evaluation using ATAM and QuaDAI

Software Architecture Evaluation using ATAM and QuaDAI

QuaDAI in O1 QuaDAI in O2 QuaDAI Questionnaire ATAM in O2 ATAM in O1 ATAM Questionnaire

ATAM in O1 ATAM in O2 ATAM Questionnaire QuaDAI in O2 QuaDAI in O1 QuaDAI Questionnaire

Software Architecture Evaluation using ATAM and QuaDAI session (60 + 60 minutes) 2nd

One of the subjects did not complete the 2nd session it was necessary to eliminate his first exercise Then, we had 30 subject distributed in 4 groups, we needed to discard two subjects, selected randomly, to maintain the balanced design consisting of a total of 28 subjects, 7 in each group

21

Validation: Results 1/3

Boxplots containing the distribution of each dependent variable

22

Validation: Results 2/3

Results, obtained through descriptive statistics, lead us to interpret that QuaDAI was more effective and efficient, and perceived as being easier to use, more useful and more likely to be used by the subjects than ATAM
Effectiveness Efficiency Duration(min)

Mean
QuaDAI ATAM
0.68 0.63

Std. Dev.
0.39 0.36

Mean
0.029 0.020

Std. Dev.
0.018 0.013

Mean
25.36 31.11

Std. Dev.
7.26 9.15

PEOU Mean QuaDAI ATAM

3.98 3.50

PU Mean
3.80 3.72

ITU Std. Dev.

0.83 0.73

Std. Dev.
0.88 0.82

Mean
3.65 3.55

Std. Dev.
0.84 0.70

23

Validation: Results 3/3

To test the statistical significance, the Mann-Whitney test were performed:
Effectiveness 0.906 PEOU 0.030 PU 0.941

ITU 0.767
Efficiency (not normally distributed): p-value from the 1-tailed t-test was 0.015

These results let us to conclude that: the difference in terms of Efficiency and PEOU was statistically significant the difference in terms of Effectiveness, PU and ITU was not statistically significant (although QuaDAI achieved better results)

24

Validation: Threats to validity: internal validity

Main threats to internal validity are: Learning effect. 2 different experimental objects, each subject applied each method with different objects, and all the possible combinations of both the method order and experimental objects

Subjects experience. No differences in the subjects experience (none had experience in architecture evaluations)
Information exchange. Different experimental objects and monitoring the subjects while they performed the tasks. The participants returned the material at the end of each session Understandability of the material. Clearing up all the misunderstandings that appeared in each experimental session

25

Validation: Threats to validity: external validity

Main threat: Representativeness of the results May be affected by the evaluation design and the participants selected The evaluation design could impact on the results due to the architectural models and quality attributes evaluated

2 different architectures, from 2 different domains, 2 different opposed NFRs and 4 different patterns for each experimental object
The participants had with no previous experience in architectural evaluations, and received only limited training on the evaluation methods

last year students (considered as novice users of architectural evaluation methods). Results could be considered as representative of novice evaluators

26

Validation: Threats to validity: construct and conclusion validity

Main threats to construct validity:

Measures applied in the analysis:

Euclidean distance has been used to measure the goodness of a solution with regard to a set of NFRs. The subjective variables are based on the Technology Acceptance Method (TAM) a model for the evaluation of information technologies.

Reliability of the questionnaire:

The reliability of the questionnaire was tested with the Cronbach test. PEOU, PU and ITU obtained Cronbachs alphas of 0.824, 0.870 and 0.831 higher than the acceptable minimum required (0.7) The main threat to conclusion validity is: Validity of the Statistical Tests Applied: This threat was alleviated by applying a set of commonly accepted tests employed in the empirical software engineering community

27

Conclusions
We presented the QuaDAI method for the derivation, evaluation and improvement of product architectures
Model-driven derivation, evaluation and improvement of product architectures Provides a systematic mechanism for dealing with those cases where the built-in variation mechanisms of the PL architecture is not enough Reuse of the architectural knowledge stored in the multimodel to help designers to decide and apply the architectural patterns required to improve product architectures

We also reported the validation of our method by means of a controlled experiment in which QuaDAI were compared ATAM
QuaDAI was more efficient and perceived to be easier to use than ATAM

With regard to the effectiveness, PU and ITU, the differences were not statistically significant (although QuaDAI achieved better results)
This may be due to the lack of experience of the subjects in architecture evaluation

28

Future work
To characterize those cases in which the variability mechanisms are not sufficient to achieve desired quality levels for a given product architecture To study other mechanisms for managing interrelationships among viewpoints in the multimodel we are using only simple values but they may not capture the full range of realworld impact relationships

we will explore the definition of functions that could express conditions on such values
To conduct replications of this experiment by considering: larger number of subjects with different subject profiles (e.g., practitioners or students with a higher level of knowledge and skills on architecture evaluation) different experimental objects in order to improve the representativeness of our results

29

Thanks for your attention!

Defining and Validating a Multimodel Approach for Product Architecture Derivation and Improvement
Javier Gonzlez-Huerta, Emilio Insfran, Silvia Abraho
ISSI Research Group, Universitat Politcnica de Valncia Camino de Vera, s/n, 46022, Valencia, Spain
{jagonzalez, einsfran, sabrahao}@dsic.upv.es

Validation: Experiment planning. Variables

The Euclidean distance, which is the ordinary distance of two points in a plane, was measured applying the formulas:
() = 1 , ( (, ) ()

, (

, =
=1

The subjective variables were measured by using a Likert scale questionnaire with a set of specific closed questions
The aggregated value of each subjective variable was calculated as the mean of the answers to the variable-related questions

31