This page outlines the practical and theoretical considerations behind .CQ, providing you with further details about the ways in which it can improve software maintainability in practice. Here you can learn more about .CQ, what the service is designed to do, and about some of the practical problems that it is aimed at.
The first part (General and Introductory Considerations) gives a broad overview over the topic of software maintenance in general, over the most common approaches to handle this issue, and some of their shortcomings.
Chapter 2 (Maintainability in Practice) looks at some typical challenges that companies are faced with when dealing with software quality issues in a practical context, at the most common ways to address them, and at some problems that regularly occur along the way. As a result, some conclusions are drawn from these observations and the most urgent requirements are listed that an alternative solution would have to fulfill.
Finally, the third section (Introducing .CQ) then presents and describes the .CQ web service, which is designed from ground up to address these issues.
If you'd like to comment on one or more of the points made below or have your own experience to add, please feel free to do so on this page's comment section. You may also discuss specific aspects with other people in one of the forums.
If you'd prefer to quickly learn about a particular element of .CQ, or have a specific question in mind, you may also try the FAQ page. To get an overall impression of what .CQ is and what it does, please take a look at the sample report.
(Disclaimer: Numbers and percentages in this text are deliberately left without direct citations, since the concrete numbers presented in research literature on software development vary widely, due to different concepts and methods. Therefore, the numbers are based on both an overall impression of the consulted research literature and on practical field experience.)
On its most basic level, the software development process can be divided into two major phases:
As you can may conclude from the above information, if software maintenance constitutes such a big proportion of a software project's budget, then any improvement that makes maintenance activities easier to accomplish will have a significantly positive impact on the project's overall costs and lifetime; Or, in other words: Improvements on the maintainability of a software system will considerably and disproportionately raise the project's Return on investment.
One common perception pertaining to software maintenance is that it merely involves fixing defects. However, in reality, such activities only account for approximately 20% of the total maintenance effort, with most of the overall “maintenance“ (around 80%) actually contributing toward non-corrective actions. A common categorization of maintenance activities divides them into the following four classes:
A well known and widely accepted concept for categorizing software maintainability is the ISO/IEC 9126 Software engineering — Product quality standard, which mentions maintainability as one of the six main characteristics of software product quality, along with analyzability, changeability, stability, and testability as its sub characteristics. However, for practitioners, the standard is of very limited use. It may provide a useful conceptual framework for discussing software maintainability, but it does not promote any concrete conclusions. In particular, it gives no clue whatsoever as to how the above sub characteristics may be correlated with observable and quantifiable code properties. Moreover, since ISO/IEC 9126 is based solely on the observation of post-release activities, it does not embody, or even imply, any element of predictability.
Turning things around, seeing them more from a developer's view, is at the heart of identifying Technical Debt that resides within a certain area of source code. What this basically means, is that there is a critical deficiency in the code properties that are known to impact maintainability, raising the likelihood of future problems. A piece of code with a high technical debt is often referred to as having a Code smell.
Whereas the ISO/IEC 9126 standard is aimed more at conceptual aspects, the Maintainability Index (MI) represents a prominent attempt to measure and quantify the maintainability of a source code body (various concrete implementations exist). On first sight, MI may be more practicable, since it calculates a concrete, single number. However, this number is the result of a pretty complicated aggregation of various different source code metrics, which are factored by some preset numbers. These, in turn, are derived from large data collections on software projects and/or expert opinions. Consequently, the formulas are very hard to understand and explain and they appear to contain an element of arbitrariness. Furthermore, since the MI represents a composite number, it tells you nothing about the concrete cause of a specific value. It is not immediately clear what change in a system causes a change in the MI metric, and therefore it is also not clear what steps can be taken to rectify any problems. In other words: The MI does not provide a mechanism for root-cause analysis. Its outcomes are not directly actionable without further interpretation, which would require additional expertise in code quality analysis or at least a considerable amount of professional experience in the field of software development.
On the business side, the Cost of (Software) Quality Approach (CoSQ) is commonly used for quantifying the expenses and savings related to software maintenance. While sometimes misinterpreted as being solely focused on reducing costs, CoSQ rather is an accounting technique that looks at the entirety of cost factors involved in delivering good-quality software. Various concrete definitions and measurement methods exist, but the general idea is always: The total costs of quality of a software system are made up of investments in good quality (i.e. any kind of quality improvement activities, mostly during the pre-release phase), plus the costs incurred as a consequence of poor quality (caused by defect repairs, remediations, and related costs that occur in the post-release phase). And - again, regardless of definitions and measurement methods - the overall result is always: Improving the good quality of a software product results in a greatly disproportionate decline on the poor quality side of the CoSQ equation. In other words: The Return on Investment for software process improvements is extraordinarily high. It usually lies between 4 and 10, with an average around 7 (meaning that every invested dollar brings 7 dollars’ profit)*. You may have heard the saying that software quality comes for free. Well, this is not correct. In fact, software quality pays. Massively. But while, at least in theory, this fact is very well known and confirmed again and again, little happens on the practical side of things. This is mostly due to the fact that, as of now, concrete implementations of software quality improvements and related controlling mechanisms are very complex undertakings, and they often require huge up-front investments. This places a big hurdle on a more widespread adoption of systematic software quality control mechanisms - especially for smaller companies.
* You may notice that our Cost Calculator
yields results that are mostly lower than these figures. This is largely due to
two facts:
1. Whereas the here described CoSQ approach looks at all aspects of a software project
- including all kind of organizational and management aspects -, .CQ focuses exclusively
on code quality at source code level and does not deal with other optimization potentials.
They might be huge, but they are not elusive for automation.
2. The Calculator results are strictly on the conservative side of things (and therefore
they are best interpreted as a kind of 'at-least' scenario).
Software development projects have a very poor industrial record*: most of them fall short of their initial objectives for one reason or the other. A good portion of them (about 5-15%) will be completely abandoned, while the majority of the rest will come in over schedule and/or over budget, mostly to a considerable extent. In reality, very few IT projects truly succeed—only about 1/4. The top reasons for this dramatically high failure rate are commonly related to poor management and planning; however, poor code quality also has a substantial share...
* While concrete figures vary largely between individual studies and occasionally are subject to heated debates (such as those presented in the infamous Standish CHAOS report), it remains true in all cases that performance in this area is generally somewhere between very poor and catastrophic.
It's always difficult to assess the value of a software system. This is mainly due to the fact that software is an intangible product and, as such, its value cannot be observed in a straightforward manner. In current practice, the overall business value of a software system tends to be overrated because its future maintenance costs are mostly undervalued. As the maintenance costs of a software system are, to a large extent, determined by the quality of its source code, a more appropriate assessment method must study this area in more detail. For this purpose, better tools and methods are needed to analyze a body of code and subsequently devise quantifiable and comparable results.
Additionally, the difficulties associated with assessing a software project's value (presumably on a regular basis), lead to a related, but distinct problem: The lack of availability of hard data makes controlling the strategic and financial elements of the project largely a matter of gut feeling (or of professional experience, if that’s how you would prefer to describe it).
Readability and understandability appear to be very trivial and, as such, their importance is significantly undervalued. It is well known that code reading and understanding consumes about half of the overall maintenance efforts. Consequently - taking the above figures into account - code reading and understanding accounts for about 25-45% of the overall project costs. While software readability is an issue in itself and partly goes beyond the code properties discussed here, the code metrics defined below are at the very heart of readability and have been proven to have a big impact on readability.
In current industrial practice, there are basically two different ways in which a company can tackle software quality issues: Through the usage of suitable tools or by hiring the services of external consultants.
There are a lot of tools available that are related to software quality issues: open source, free and commercial (e.g. StyleCop, FxCop, or NDepend in the .NET space, to name just some of the more popular offerings). But their adoption in an industrial-grade context faces some serious issues that greatly limit their practical usefulness: It's not easy to integrate them with a companies' toolchain and to align them with the developer's familiar workflows. Additionally, the reports that they output are exclusively directed towards “nerdy” software developers, more often than not require very specialized expertise about code quality issues and generally present an incomprehensible pile of details about the inspected code while at the same time lacking a reader-friendly summary.
An alternative approach is to hire an external consultant who has the required expertise to provide product advice on methods of improving a software system's code quality. However, the availability of such experts is very limited. Furthermore, because these experts usually charge high rates, hiring the right person can constitute a costly initiative. Moreover, these types of consultants don’t naturally contribute toward the establishment of the required in-house expertise within a company.
Both approaches—tool adoption and the assignment of external consultants—come with substantial downsides that effectively prevent their large-scale adoption in the industry. They both require massive up-front investments (both in terms of money and effort) and their effects can only be observed in practice after a considerable period of time (if at all). There is a distinct possibility that an investment in this area will never actually pay off in the final settlement - especially if their initial assumptions were not chosen carefully (which sometimes is very close to impossible...) or there were significant changes to the project (which is the case all too often...). There's not much room for trial and error and/or flexibility here. Consequently, most people who are in charge of a software project's budget will not even consider taking such a high business risk.
The previous sections described some of the major practical problems that can arise when a company or a development team attempt to resolve software maintainability or code quality issues.
Below you will find a list of the main principles and properties that a suitable and user-oriented solution will need to demonstrate in order to respond to these problems. Namely, such a solution should...
The above considerations led to the conceptual design of a solution named .CQ. .CQ is a cloud-based web service that analyzes a software project's source code and presents the results of this analysis in a suitable report document. The following sections briefly outline the principles that .CQ is built upon, and how this addresses the mentioned issues.
An analysis of the maintainability of a software project first and foremost has to quantitatively measure the most relevant properties of the source code under question. These measurements are commonly known as software metrics. A small but significant set of these metrics is used to analyze the project's source code; each of the chosen metrics has a well-established record of scientific verification and corresponding expert judgments. Namely, these metrics are:
Though not allowing for direct root-cause analysis, it is fairly intuitive that the total size of a software system has a big impact on its overall maintainability. The larger a system is, the more effort it takes to maintain it, since there is more information to be taken into account. System volume is measured in Source lines of code (SLOC).
The complexity of source code refers to its internal intricacy and can be measured using the McCabe Cyclomatic Complexity metric, which counts the number of possible decisions that can be taken in a procedure. The complexity of a unit negatively impacts its analyzability and testability—more complex units are more difficult to understand and analyze, more difficult to test, and they tend to contain more defects. Therefore, it is advisable to make every effort to keep a method's complexity low.
Duplicate code is another code smell that is well known to have a substantially negative impact on a software system's maintainability. Although the occurrence of a small amount of duplication is natural and unavoidable in complex systems, excessive amounts of duplication are detrimental to its maintainability. Sequences of duplicate code are also referred to as code clones (or simply clones).
Large methods are hard to read and understand and also tend to cause more issues than smaller ones. As such, it is commonly accepted among developers that methods should be kept small and focused. Just like overall system volume, method size is expressed in Source lines of code (SLOC).
Each of the above individual metrics can be challenged, do not provide hard enough data when viewed in isolation, and may, in some cases, yield false positives. For this reason, the individual metrics on the methods are combined to a summarized rating that produces more authoritative results. Additionally, the observed metrics are mapped to maintainability characteristics as defined by the ISO/IEC 9126 standard (described above). The methodology employed is based on a modification of the software quality assessment model known as the SIG maintainability model.
.CQ is intended to be repeatedly applied. This can either be on a regular basis during the development phase of a software project (maybe as part of a Continuous Integration process) or on a per-case basis for existing source code bodies (e.g. for legacy or outsourced projects). This way, a data inventory for the project can be compiled that offers deeper insights into the historical trends of the analyzed system. This is especially important because synchronous analysis results can vary greatly between projects, meaning that their significance and comparability are limited. On the other hand, historical data that only relates to previous data from the same project has much greater explanatory power.
To really control business processes and to assess specific actions, it is important that it is possible to relate the analysis figures to more business and project-specific data—be it bug frequency, sales figures, or any other data type that is meaningful in a certain scenario. To make this possible, the user can define custom data series and enter the respective figures. The .CQ service then computes the statistical correlation between the user-defined data and the system's maintainability rating, using Spearman's rank correlation coefficient.
The results of the analysis are relevant to two different groups of people, performing different actions, and looking at different figures:
An analysis report that strives to be useful to both groups must provide detailed information about the source code to allow for direct action while also presenting the data in a more summarized form that is useful for business. (See here for an example report.)
.CQ's architectural approach is that of a (RESTful) web API. This design brings a great amount of flexibility, making it possible to build any sort of client application on top of the service (e.g. command-line clients or IDE-plugins).
The following is a compilation of sources that influenced the design of the .CQ concept. It includes blogs, whitepapers, research studies and books. Where freely available, these links point to the respective sources. Other links point to Amazon (books) or the respective locations where the research publications are available.
Abran, A. (ed), Moore, J.W. (ed), Bourque, P. (ed), Dupuis, R. (ed) . 2005. Guide to the Software Engineering Body of Knowledge (SWEBOK), Chapter 6: Software Maintenance. IEEE Computer Society.
Agarwal, R. 2005. Software Development vs Software Maintenance.
Aggarwal, K.K, Singh, Y., Chhabra, J.K. 2002. An integrated measure of software maintainability. Annual Reliability and Maintainability Conference..
Anil kumar, G., Reddy, CRK., Govardhan, A. 2013. Influences of code cloning in software maintenance cost. In: International Journal of Advanced Trends in Computer Science and Engineering, Vol.2 , No.1, pp. 640-643.
Atwood, J., 2006. The Long, Dismal History of Software Project Failure. Coding Horror.
Atwood, J., 2004. The Cost of Complexity. Coding Horror.
Banker, R.D., Datar, S.M., Kemerer, C.F., Zweig, D. 1993. Software Complexity and Maintenance Costs. In: Communications of the ACM, Vol. 36, Issue 11, pp. 81-94.
Bird, J. 2013. Classic Mistakes in Software Development and Maintenance.
Bullard, R. The Maintenance Phase in the Software Life Cycle. eHow.
Boehm, B. 1981. Software Engineering Economics. Prentice Hall.
Bonett, D.B., Wright, T.A. 2000. Sample size requirements for estimating pearson, kendall and spearman correlations. In: Psychometrika, Vol. 65, Issue 1, pp. 23-28.
Charette, R.N. 2005. Why Software Fails. IEEE Spectrum.
Chhabra, J.K. 2011. Code Cognitive Complexity: A New Measure. In: Proceedings of the World Congress on Engineering 2011 Vol 2.
Cognence Inc. 2005. Are Software Quality Problems Costing Your Company?. Whitepaper.
Cordy, J.R. 2003. Comprehending Reality - Practical Barriers to Industrial Adoption of Software Maintenance Automation. In: Proceedings of the 11th IEEE International Workshop on Program Comprehension, p. 196.
Dutta, S. 2007. Recognising the True Value of Software Assets. Insead.
El Emam, K. 2005. The ROI from Software Quality. Auerbach Publications.
Eveleens, J.L., Verhoef C. 2010. The rise and fall of the Chaos report figures. In: IEEE Software archive, Volume 27, Issue 1, pp. 30-36.
Fowler, M., Beck, K., Brant, J., Opdyke, W., Roberts, D. 1999. Refactoring: Improving the Design of Existing Code. Addison-Wesley.
Galorath, D. 2010. More Software Project Failure / Challenge Information From CAI. Dan on Estimating.
Galorath, D. 2012. Software Project Failure Costs Billions.. Better Estimation & Planning Can Help. Dan on Estimating.
Garcia, M.J.B., Alvarez, J.C.G. 1996. Maintainability as a key factor in maintenance productivity: a case study. In: Proceedings of the 1996 International Conference on Software Maintenance, p. 87.
Glass, R.L. 2001. Frequently Forgotten Fundamental Facts about Software Engineering. In: Journal IEEE Software Vol. 18 Issue 3, p. 112.
Glass, R.L. 2005. IT Failure Rates—70% or 10–15%? In: IEEE Software archive, Vol. 22, Issue 3, pp. 110-112.
Glass, R.L. 2006. The Standish Report: Does It Really Describe a Software Crisis? In: Communications of the ACM, Vol. 49, No. 8, pp.15-16.
de Groot, J., Nugroho, A., Back, T., Visser, J. 2012. What Is the Value of Your Software?. In: 3rd International Workshop on Managing Technical Debt (MTD), pp. 37-44.
Heitlager, I.,Kuipers, T., Visser, J. 2007. A Practical Model for Measuring Maintainability. In: Proceedings of the 6th International Conference on Quality of Information and Communications Technology, pp. 30-39.
Jones, C. 2006. The Economics of Software Maintenance in the Twenty First Century. Computer Aid.
Jones, C. 2008. Applied Software Measurement: Global Analysis of Productivity and Quality. McGraw-Hill.
Kapser, C., Godfrey, M.W. 2006. “Cloning considered harmful” considered harmful: patterns of cloning in software. In: 13th Working Conference on Reverse Engineering (WCRE 2006). IEEE Computer Society, pp. 19–28.
Karg, L.M., Grottke, M., Beckhaus, A. 2011. A Systematic Literature Review of Software Quality Cost Research. In: Journal of Systems and Software, Vol. 84, No. 3, pp. 415-427.
Krasner, H.. 1998. Using the Cost of Quality Approach for Software. In: Crosstalk, The Journal of Defense Software Engineering, pp. 6-11
Laitinen, K. 1996. Estimating Understandability of Software Documents. In: ACM SIGSOFT Software Engineering Notes, Vol. 21, Issue 4, pp 81-92.
Laporte, C.Y., Berrhouma, N., Doucet, M., Palza-Vargas, E. 2012. Measuring the Cost of Software Quality of a Large Software Project at Bombardier Transportation: A Case Study. In: Software Quality Professional, Vol. 14, Issue 3, pp. 14-31.
Leach, R. J. 2000. Introduction to software engineering. Crc Press.
Luijten B., Visser, J. 2010. Faster Defect Resolution with Higher Technical Quality of Software. In: Proc. of the 4th International Workshop on System Quality and Maintainability, pp. 11-20.
McCabe, T.J. 1976. A Complexity Measure. In: IEEE Transactions on Software Engineering Vol. 2, No. 4, p. 308.
McConnell, S. 2004. Code Complete: A Practical Handbook of Software Construction, Second Edition.. Microsoft Press.
McKenna, P. 2005. Assessing the economic value of software projects. IBM.
Mead, N.R., Allen, J.H., Conklin, W.A., Drommi, A., Harrison, J., Ingalsbe, J., Rainey, J., Shoemaker, D. 2009. Making the Business Case for Software Assurance. Software Engineering Institute Special Report CMU/SEI-2009-SR-001.
Mierlitz, L. 2012. Gartner Survey Shows Why Projects Fail. thisiswhatgoodlookslike.
Newton, C. 2010. Software maintenance. Clarity in Code.
Nugroho, A., Kuipers, T., Visser, J. 2011. An Empirical Model of Technical Debt and Interest. In: Proceedings of the 2nd Workshop on Managing Technical Debt, pp. 1-8.
Omnext. 2010. How to save on software maintenance costs. Whitepaper.
Pigoski, T.M. 1996. Practical Software Maintenance. John Wiley & Sons.
Roberts, M.. 2012. Cost of Quality: More than Risk and Compliance. LNS Research.
Salado, J. 2012. 6 best practices to save in software maintenance. Optimyth.
Seacord, R.C., Plakosh, D., Lewis, G.A. 2003. Modernizing Legacy Systems: Software Technologies, Engineering Processes, and Business Practices. Addison-Wesley.
Sessions, R. 2009. The IT Complexity Crisis: Danger and Opportunity.
Sharpe, R. 2008. McCabe Cyclomatic Complexity: the proof in the pudding. Enerjy.
Sims, C. 2010. Estimating Business Value. InfoQ.
SolidSourceIT. 2012. Does source code duplication matter?
Stafford, P. 2003. Software Maintenance As Part of the Software Life Cycle. Tufts University (Boston, MA), Department of Computer Science.
The Economist. 2004. Make it simple.
The Economist. 2004. Keep it simple.
Table of Critical Values of the Spearman Rank Order Correlation Coefficients.
van Solingen, R. 2004. Measuring the ROI of Software Process Improvement. In: IEEE Software archive, Vol. 21, Issue 3, pp. 32-38.
Zar, J.H. 1972. Significance Testing of the Spearman Rank Correlation Coefficient. In: Journal of the American Statistical Association Vol. 67, No. 339, pp. 578-580.
blog comments powered by Disqus