JOURNAL OF COMPUTER SCIENCE AND ENGINEERING, VOLUME 13, ISSUE 2, JUNE 2012 10

Software Quality Framework

Malik F. Saleh
Abstract In this paper we present a general software quality model, providing the possibility to describe different concepts related to quality. We show that our quality model is able to integrate the various concepts found in quality standards and different quality models. Furthermore, we provide different quality views related to software quality, enabling consistency and continuity of quality-related information. The most influential factor for the developers of software is the customer perception. We connect the developer with the customer to derive a common interpretation for quality. This paper introduces a model for software quality by connecting and integrating the different views of software quality. In addition, it connects the customer view with the developer view of software quality and it treats software as a product. Index Terms Software quality, quality framework, software engineering, software assurance.

1 INTRODUCTION

roducing software with high quality is a challenging task. Many software projects fail before they are completed while other projects fail after deployment. If you are developing software then quality is often on your mind. But what does software quality really mean? When users encounter errors in software applications, the general public of users blames the errors on the system making no distinction between hardware, software or user input. Are developers to be blamed for all aspect of software quality? Assuring high quality of software is crucial. The disciplines of software engineering are all aiming to produce software that assures quality [1]. Although quality is crucial, it is a highly complex topic which required the disciplines of software engineering to develop their own quality assurance approaches. Many quality metrics have been proposed. Various metrics can be proposed to evaluate the way the quality is affected [2] and many are being used such as the number of lines of code (LOC). In fact, many companies have built their own quality frameworks based on product metrics. [3]. Quality standards were created such as ISO standards. Software metrics have been proved to reflect software quality, and thus have been widely used in software quality evaluation methods [2]. The customer plays an important role in software quality. Many definitions of software quality have been published. In general, most definitions include the phrase customer satisfaction [4, 5]. According to ISO 8402, quality is the total characteristics of an entity that bear on its ability to satisfy stated and implied needs. ISO/IEC 9126 defines a quality model that comprises six characteristics of software product quality. These characteristics are: Functionality, Reliability, Usability, Efficiency, Maintainability, and Portability. This quality model is generic and can be ap

plied to any software product by tailoring it to a specific purpose [5]. In this paper we present a general software quality model, while describing very different concepts related to quality. We show that our quality model can integrate various characteristics found in quality standards and different quality models. Furthermore, we show that this quality model is able to describe the interrelations of disciplines, like requirements engineering and software testing, to software quality. With this quality model, we provide different views common in concepts related to software quality, enabling consistency and continuity of quality-related information.

2 BACKGROUND AND PROBLEM STATEMENT

Different approaches have been developed to assure software quality. A good definition of software quality must measure quality in a meaningful way. This measure should consider all stages of the software product such as software development, distribution, manufacturing, the user experience, and maintenance. The literature points out many of the problems that software quality faces: There is no comprehensive framework to describe all different concepts relating to software quality in a common way [1]. There does not exist a rigorous method for measuring customer perception of product quality [4]. Surveys of customer opinion cannot be considered as measurements, since they are based on non-objective assignments that vary according to user judgments. A measure, however, should be an empirical assignment of a number to an entity to characterize its specific attribute. There are many requirements that affect the quality of software. For example operation require-

M.F. Saleh is with Prince Mohammd Bin Fahd University, P.O Box 1664 Al Khobar 31952 Saudi Arabia.

2012 JCSE www.journalcse.co.uk

11

ment measure the software functionalities without the necessary non-functional requirements. Therefore, the quality characteristics are treated as technical issues related to the design and testing of the software product [6]. While the product revision requirement is affected by how the software will be changed. It is determined by how easily bugs can be found and corrected and the efforts needed to modify the operational software. Finally, software portability is determined by how easily the software can be used on different platforms and it is determined by its reusability factor. Quality in software projects is discussed in the context of tolerated errors in software and the meanings of "safety-critical" and "missioncritical". In addition, software quality is discussed in the context of software contribution to the larger functionality and the quality of products and services [7].

the two methods dont represent a quantifiable quality measurement The developer view of software quality is different from the customer view. For example the customer interprets the quality of operation as meeting the requirement while the developers are using different factors. The developer view of quality in the software is affected by many factors. This model stresses on three primary ones: 1. The code - It is measured by its correctness and reliability 2. The data - It is measured by the application integrity 3. Maintainability it has different measures the simplest is the Mean Time To Change

These problems are rather unconnected and not integrated. They lack a common foundation and they lack a common quality measure, which leads to loss of quality. Finally, there is no comprehensive framework to describe all the different views of concepts relating to software quality.

SOFTWARE QUALITY FRAMEWORK

3.2.1 Correctness and Reliability Correctness and Reliability determine how well a system fulfils the customers overall objectives; measuring correctness involves keeping track of reported defects and using a simple quantitative approach. The defects per thousand lines of code (KLOC) can be used by the developers to track the number of lines affected by defects when making correction to the code. The percentage of the affected lines to the total lines of code is the measured to be used. However, this measure is almost impossible to interpret alone. It must be complemented by the measure of functional point analysis.
Errors that may appear to be difficult are found to be simple when dissected into their components and placed into their appropriate categories. Function Point Analysis (FPA) is a method to break systems into smaller components, so they can be better understood and analyzed [8]. Developers need to combine both the KLOC and FPA to analyze the correctness of the code. FPA is used to produce functional decomposition of software and it can be used to size software applications. Combining both measures will derive a more accurate measure than just using the KLOC alone. Correcting errors also involves using both measures. For example, an application that consists of one million lines of code is broken down into twenty function points. The average for each function point is 100 thousands line of code. If developers recorded errors that affected 1000 lines per error in five function points then the measure is as follow:

This paper introduces a model for software quality by connecting and integrating the different views of software quality. This model connects the customer view with the developer view of software quality and it treats software as a product. The software product view reviews the characteristics of a product that bear on its ability to satisfy stated and implied needs. This model is also a framework that describes all the different concepts relating to quality in a common way measured by qualitative scale that can be understood and interpreted in a common way. Therefore the most influential factor for the developers is the customer perception. This model connects the developer with the customer to derive a common interpretation for quality.

3.1 Developers View

Validation and verification are two independent methods used together for checking that a software product meets the requirements and that it fulfills its intended purpose. Validation checks that the product design satisfies the intended usage and verification checks for errors in the software. The primary concern for developers is in the design and engineering processes involved in producing software. Quality can be measured by the degree of conformance to predetermined requirement and standards, and deviations from these standards can lead to poor quality and low reliability. While validation and verification are used by the developers to improve the software,

Table 1
Measuring Correctness using FPA and LOC No. LOC Per No. LOC Error FP Error Affected 1 100,000 1 1000 1% 2 100,000 2 2000 2% 3 100,000 3 3000 3% 4 100,000 4 4000 4% 5 100,000 5 5000 5% 1 Million 15 Er- 15K 1.5% rors

Function Points

12

Average Errors

3%

Therefore, the average correctness for the affected function points is 97. The overall correctness for the software is: 100% - 1.5% = 98.5% This method has many advantages: a. Finding the strength and weaknesses in the application b. Defining what to re-engineer c. Estimating the overall effort and schedule of maintenance 3.1.2 Integrity It is measured by considering the proportion of attacks as opposed to valid uses of the software. A quantitative measure is used by grouping the same types of attacks together and assigning a value ranging from 1 (Perfect) to 0 (weakness) to the attack group. The following measure is used to derive the overall Integrity of the software:

0.01 Attempted 20 / failed lo- 1000 = gon 0.02 Total

16/20 = 0.8

= 0.792 (1 0.02) * (1 0.8) = 0.196 1.18

Therefore, the overall integrity for the software is: 100% 1.18% = 98.82 Integrity. 3.1.3 Maintainability

It determines how easily bugs can be found and fixed. There is no clear way of measuring maintainability, but it can be seen as the ease with which the software can be corrected or perfected. A simple measure is mean time to change (MTTC) [9, 10]. Mean time to change requires keeping track of reported bugs. It is the average of the time it takes to analyze a reported bug, design the correct modification, implement the change, test the software, and distribute the change to the customer. This quality factor affects user perception of the software. The lower the mean time the more maintainable the software product is perceived by the user

(1) (2)

3.2 User View

The software integrity is therefore measured by grouping all types of integrity attacks

(3)

For example, an application has one thousand entries for user access in the log file. Of these, one hundred of the en- tries were attacks, grouped as follow:

50 Denial of Service (DOS) attacks. Five successful DOS attacks 20 password guessing attacks. Two successful attacks 10 accidental attacks. Two successful attacks 20 failed logon. Four accounts were locked

Table 2
No. Active Attacks 1 Measuring Integrity Type Threat Security Denial of Service Password Guessing Accidental attacks 50 /1000 = 0.05 20 /1000 = 0.02 10 / 1000 = 45/50 = 0.90 18/20 = 0.9 2 / 10 = 0.2 Integrity
attacks

The highest quality in software is expected by the users when they acquire software. The quality is different when end users develop their software. End-user programming, a phrase popularized by [11] which is programming to achieve the result of a program primarily for personal, rather than public use. The important distinction here is that software itself is not primarily intended for use by a large number of users with varying needs. For example, a teacher may write a grades spreadsheet to track students test scores. In these end-user programming situations, the program is a means to an end that could be used to accomplish a goal. In contrast to end-user programming, professional programming has the goal of producing software for others to use. For example, the moment a novice Web developer moves from designing a web page for himself to designing a Web page for others, the nature of this activity has changed. The moment this shift in intent occurs, the developer must plan and design for a broader range of possible uses, increasing the importance of design and testing, and the prevalence of potential bugs [12] therefore, increasing the importance of the software quality.

Users perceive software quality as a fit between their (1-0.05) * goals and software's functionality. The better the quality, (1 0.90) the more likely the user will be satisfied with the soft= 0.095 (1 0.02) ware. When the quality is bad, developers must meet user * (1 0.9) needs or face a diminishing demand for their software [13]. Therefore, the user sees quality as fitness for purpose = 0.098 [7]. Avoiding complexity and keeping software simple, (1 0.01) considerably lessens the implementation risk of software. * (1 0.2) In some instances, users abandon the implementation of

13

complex software because the software developers were expecting the users to change their business and to go with the way the software works [14]. Measuring quality of the software is therefore determined by whether the software is accomplishing the goal for the user and the number of errors or unexpected behavior of the software that the users face during the software normal use. Complex systems inherently present unique risks due to tightly linked interdependencies of business processes. Users need to have an understanding of complexity risks when planning and conducting assurance of the reliability of these complex software [15]. Prototyping can help simplify the complexity of the software. Prototyping is a critical component in evolutionary development [16]. Prototyping is recommended for clarity in understanding system requirements and in planning systems architecture [17] and thus can help in improving software quality. The user experience with the software can be a momentary experience. Hassenzahl (2008) argued that the user experience is primarily an evaluation of feelings while interacting with a product or service. By that, the user experience shifts attention from the product and materials to humans and feelings and it changes over time. Tracking the user experience, therefore, is very challenging. Tracking the user attitude toward the software requires evaluating the software while the user interacts with the software and this attitude can change from time to time. Therefore, the user experience should not be evaluated based on a single outstanding moments and it should be done continually over different time periods [16]. A simple survey can be used to capture the perceived user view of the quality of the software. Appendix 1 lists some questions that quantify the answers. The items in this survey were adopted from [14]. These items were also used by [18] and by [19]. Each item was scored using a five-point scale ranging from never (1) to always (5). All items were presented such that the greater the score, the greater the satisfaction of the particular item

4 CONTRIBUTION AND CONCLUSION

A well-rounded model was introduced that covered the three views of software, the first view dealt with the inception and development of software by the developers. The second view dealt with the customer when using the software. The final view treated the software as a product. This framework handled the perception of quality in a manner that quantifies quality into measurable indicators. This perception of quality was comprehensive and it describes all different concepts relating to software quality in a common way that can be understood by the numbers. Customer survey was introduced for measuring customer perception of product quality. Surveys of customer opinion were empirical by assigning a number to an entity to characterize its specific attribute. This survey measures the perception of the customer, which changes over time. Therefore, the survey is repeated many times. This model examined the different requirements that affect the quality of software including the technical, nontechnical requirement, and the software revision requirement. The technical requirement and the revision requirement were measured in the developer view while the non-technical requirements were measured in the customer view. Finally, this model discussed quality in the context of stakeholder (the developer and the user) satisfactions instead of the context of tolerated errors in software. In addition, software quality is discussed in the context of software contribution to the larger functionality and the quality of products.

APPENDIX A
Customer Satisfaction Survey The scales ask you to indicate whether these practices are satisfactory and to show how well this is done. Use a five-point scale ranging from never (1) to always (5). Description
1. To what extent does the software quality as- surance function have a management report- ing channel separate from the software devel- opment project management? To what extent is a formal procedure used in the management review of each software de- velopment prior to making contractual com- mitments? To what extent is a formal procedure used to make estimates of software size? To what extent is a formal procedure used to produce software development schedules? To what extent are formal procedures applied to estimating software development costs?

Rating

3.3 Product View The product view sees quality as tied to inherent characteristics of the product. The Free Dictionary defines the product quality as the collection of features and characteristics of a product that contribute to its ability to meet given requirements [20]. Product quality can be measured by the value-based view which sees the quality as dependent on the amount a customer is willing to pay for it. To the users, a high-quality product is one that satisfies their preferences and expectations while meeting their requirement. End-users satisfaction of the product represents wiliness to learn, use, upgrade the product, and when asked to participate in rating the product, a positive rating is casted.

2.

3. 4. 5.

14

6.

To what extent are profiles of software size maintained for each software configuration item over time? To what extent are statistics on software de- sign errors gathered? To what extent are there statistics on software code and test errors? To what extent do software development first- line managers sign off on their schedules and cost estimates?

7. 8. 9.

10. To what extent is a mechanism used for con- trolling changes to software requirements? 11. To what extent is a mechanism used for con- trolling changes to the code? (Who can make changes and under what circumstances?) 12. To what extent is there a software engineering process group function? 13. To what extent is there a required software engineering training program for software de- velopers? 14. To what extent is a formal training program required for design and code review leader? 15. To what extent does the software organization use a standardized software development pro- cess? 16. To what extent does the software organization use a standardized and documented software development process on each project? 17. To what extent does senior management have a mechanism for the regular review of the sta- tus of software development projects? 18. To what extent are the action items resulting from design reviews tracked to closure? 19. To what extent are the action items resulting from code reviews tracked to closure? 20. To what extent is a mechanism used for ensur- ing compliance with the software engineering standards? 21. To what extent are internal software design reviews conducted?

22. To what extent is a mechanism used for con- trolling changes to software design? 23. To what extent are software code reviews con- ducted? 24. To what extent is a mechanism used for verify- ing that the samples examined by software quality assurance are truly representative of the work performed? 25. To what extent is a mechanism used for man- aging and supporting the introduction of new technologies? 26. To what extent is test coverage measured and recorded for each phase of functional testing? 27. To what extent are analyses of errors conduct- ed to determine their process related causes? 28. To what extent is the error data from code re- views and tests analyzed to determine the like- ly distribution and characteristics of the errors remaining in the product? 29. Keep the system simple 30. Hide complexity 31. Avoid changes 32. Obtain user participation 33. Obtain user commitment 34. Providing training to users 35. Providing on-going assistance 36. Use a prototype approach for system devel- opment 37. Tailor system to peoples capabilities 38. Reliable software 39. Efficient cost of software operations 40. Wide range of outputs that can be generated and queries that can be answered 41. Overall responsive software to users 42. Ability to meet project goals 43. Adherence to schedule

15

44. Adherence to budget

[19] Henderson, J.C. and A.S. Lee, Managing I/S design teams: a control theories perspective. . Management Science, 1992. 36(6): p. 757777 [20] TheFreeDictionary. Product quality TheFreeDictionary 2012; Available from: http://encyclopedia2.thefreedictionary.com/Product+quality Malik F. Saleh is a professot at Prince Mohamamd Bin Fahd University in the Kingdom of Saudi Arabia with over 15 years of teaching experience. Dr. Saleh Chaired the MIS department and he has a leadership role in the maintenance of academic standards and in the development of educational policy and of curriculum areas within the University.

ACKNOWLEDGMENT
The author wishes to thank Prince Mohammad Bin Fahd Univerisity for the support provided in publishing this research.

REFERENCES
[1] Lochmann, K. and A. Goeb, A unifying model for software quality, in Proceedings of the 8th international workshop on Software quality. 2011, ACM: Szeged, Hungary. p. 3-10 Stroggylos, K. and D. Spinellis, Refactoring--Does It Improve Software Quality?, in Proceedings of the 5th International Workshop on Software Quality. 2007, IEEE Computer Society. p. 10 Basili, V.R., L. Briand, and W.L. Melo, A validation of objectoriented design metrics as quality indicators. IEEE Trans. Softw. Eng., 1996. 22(10): p. 751-761 Xenos, M. and D. Christodoulakis, Measuring perceived software quality. Information and Software Technology, 1997. 39(6): p. 417-424 Jung, H.-W., S.-G. Kim, and C.-S. Chung, Measuring Software Product Quality: A Survey of ISO/IEC 9126. IEEE SOFTWARE, 2004. 21(5): p. 88-92 Chung, L., et al., Non-Functional Requirements in Software Engineering. 2000, Boston: Kluwer Academic Publishers Kitchenham, B. and S.L. Pfleeger, Software Quality: The elusive Target. IEEE Software, 1996. 13(1): p. 12-21 Longstreet, D. Function Points Analysis Training Course. 2012; Available from: http://www.softwaremetrics.com Bhatt, P., G. Shroff, and A.K. Misra, Dynamics of Software Maintenance. ACM SIGSOFT Software Engineering Notes, 2004. 29(5) Akingbehin, K. and B. Maxim. A Three-Layer Model for Software Engineering Metrics. in Proceedings of the Seventh ACIS International Conference on Software Engineering, Artificial Intelligence, Networking, and Parallel/Distributed Computing. 2006. Las Vegas, Nevada, USA Nardi, B.A., A Small Matter of Programming: Perspectives on End User Computing. . The MIT Press, 1993 Ko, A.J., et al., The State of the Art in End-User Software Engineering. ACM Computing Surveys, 2011. 43(3) Robinson, W.N., Seeking Quality through User-Goal Monitoring. IEEE Softw., 2009. 26(5): p. 58-65 Subramanian, G.H., J.J. Jiang, and G. Klein, Software quality and IS project performance improvements from software development process maturity and IS implementation strategies Journal of Systems and Software, 2007. 80(4): p. 616-627 Wright, S. and A. Wright, Information system assurance for enterprise resource planning systems: unique risk considerations. Journal of Information Systems, 2002. 16(1): p. 99113 Pressman, R., Software Engineering: A Practitioner's Approach Vol. 6. 2004: McGraw-Hill Boehm, B.W. and P.N. Papaccio, Understanding and controlling software costs. IEEE Transactions on Software Engineering, 1988. 14(10): p. 14621477 Robey, D., L.A. Smith, and L.R. Vijayasarathy, Perceptions of conflict and success in information system development projects. Journal of Management Information Systems, 1993. 10(1): p. 123139

[2]

[3]

[4]

[5]

[6] [7] [8] [9] [10]

[11] [12] [13] [14]

[15] [16] [17] [18]