Fallacy of Certifications

Murali Chemuturi

Introduction

The definition of quality (Fitness for Use) itself leaves a bit to be desired. It leaves theterms “Fitness” and “Use” open to interpretation. Thus we see a plethora of productsclaiming to be of “Quality” without even adding adjectives like “good”, or “best”. Still theterm “quality” itself implies great quality in the minds of people in general. Inmanufacturing and other engineering fields, industry associations, Governments, andArmed Forces brought out standards and conforming to those standards ensured “GoodQuality” deliverables.When it came to software, the picture of standardization is not that bright. True, IEEE(Institution of Electrical and Electronic Engineers, USA) brought out some standards andtermed them as Software Engineering Standards – not “Software (Quality) Standards”.These standards are more in the nature of guidelines rather than standards in the strictsense of the word “standard”. They are open to interpretation and adaptation.Somehow, it came to be believed that adhering to a defined process would ensuresoftware quality and ISO (International Standards Organization) brought out 9000 seriesof standards. This is followed by SEI (Software Engineering Institute, Carnegie MellonUniversity, USA) brought out CMM and CMMI (Capabilities Maturity Model Integrated).CMMI itself is twofold – CMMI for Development and CMMI for Acquisition.Many organizations wishing to outsource their development work started insisting oncertification, especially of CMMI. A certificate enables opening of doors for bidding andlack of it, closed the doors. The vendors started getting the coveted certificate either byhook or by crook, just to be in the race. And understandably certification organizationssprouted like mushrooms. Now plenty of development organizations got certified – atleast on ISO and in many cases both ISO and CMMI.We also have TMM (Testing Maturity Model), People Capability Maturity Model (PCMM),Software Engineering Capability Maturity Model (SE-CMM), IT Service CMM and so on -Lee Copelandlists

34 maturity models

in his article “The Maturity Maturity Model (M3)”on Stickyminds.com web site.ISO 9000 series of standards started out focusing on Quality with QMS (QualityManagement System) as the main document and Quality Policy as the backbone for theorganizational processes.But perhaps due to pressure from industry, these process standards diluted intoorganizational processes, shifting the focus from quality to organizational vision, andgoals etc. CMMI goes one step further stating that the process should be to achievebusiness goals. The quality of deliverables has clearly taken backseat.I had occasion to be associated with some certified organizations, as a consultant or asa member of the audit team or as an employee. A significant number of those do notadhere to their own defined process. I was horrified to find one ISO 9000 certifiedorganization in which the MR (Management Representative) did not read the processdocumentation. I noticed that the quality head of a CMM level 5 organization, doesn’t

know how to open the URL for the organizational process. I observed a CMM level 4certified organization that does not collect nor maintain any metrics. In yet in anotherorganization that is certified for ISO 9000 and aiming for CMMI level 3, I heard the CEOstating that he does not want any managers in his organization – I was dumbstruckwondering who would manage their software projects, if everybody were a coder.Organizations unearthed the loopholes in the models, consultants advise how to cookthe books to get the certificate and appraisers are available who would certify for a fee. Iposed a question for the CMMI appraisers group on Yahoo if they ever refused acertificate – only one or two replied in the affirmative – rest all maintained a dignifiedsilence.The time has come to develop a different paradigm for quality in software, a paradigmthat is focused on the quality of the deliverable than on the organization.Let us examine some of the criticisms of these Maturity Models (MMs).One criticism of all these MMs is that they emphasize on the organizational businessobjectives, but not on Product (or Deliverable if you prefer) Quality! The confidence thata process driven organization delivers Quality is misplaced. One – the process itselfmay be flawed. Two – each process has loopholes. Three – the people are focused onconforming to process, more than achieving excellence in quality. Four – managementof the organization focuses on delivering and selling than on quality. The poor qualityhead, if there is one, is there to coordinate with certifying agencies and he has no controlwhatsoever on the product quality. In many organizations, I saw that the person holdingthe post of quality head (under many other nomenclatures like SEPG Head, QualityCoordinator, Quality Manager, Director Quality and so on) is not really qualified orexperienced for holding that post or possess much knowledge about the qualityconcepts and tools.MMs focused less on development of software and ensuring that quality is built-in butfocused more on support processes. CMMI has more (8) specific goals for ProjectManagement (Project Planning, Integrated Project Management, Risk Management,Configuration Management, Project Monitoring and Control, Quantitative ProjectManagement, Supplier Agreement Management, and Requirements Management)where as it has less (3) specific goals for Quality (Process and Product QualityAssurance, Validation and Verification). It has only three specific goals (ProductIntegration, Requirements Development and Technical Solution) for development ofsoftware. It has two specific goals (Organizational Process Focus and OrganizationalProcess Development) for organizational process definition and four specific goals(Causal Analysis and Resolution, Decision Analysis and Resolution, Measurement andAnalysis, and organizational Process Performance) for measurement and analysis. Theremaining two specific goals are Organizational Innovation and Deployment andOrganizational Training. Thus the focus on Quality is too diluted – three out of 22!Even then, the MMs do not insist that their model must be implemented in to-to. Theyaccept “largely implemented” as adequate for giving a certificate. The model itself is nottightly defined and is made so flexible that the practices are open to any interpretation.Some allow “Alternative Practices” in place of the practices defined in the model. Thisallows the organization to do what they want and still get certified as conforming to themodel!

Another criticism of the MMs is that they do not define any quality thresholds forachieving the certification. Conformance to self-defined process is adequate. Say thestandard for an electrical equipment would define the insulation resistance in quantitativeterms so that human beings do not get an electric shock by handling that equipment. Butno software engineering standard defines what should be the defect density for, let ussay, a financial application!Another criticism of MMs is that they do not specify the number of years the organizationneeds to be in operation before they can be

mature

enough to be ready for certification.That way, even a one-year-old organization can get certified. There are single personorganizations that are certified!Another criticism about the MMs is that they do not specify any quality objectives to beachieved for obtaining the certificate. Mere conformance or showing evidence ofconformance in just six projects or less gets the organization certified – no need todemonstrate achievement of quality!The owners of the MMs do not maintain the actual performance of the organizationsafter certification. ISO specifies surveillance audit that is cursory but CMMI does not.Whether the quality has improved, or any reduction in complaints – the owners of MMsdo not keep track of.Let us examine some loopholes in these certifications.

Pecuniary consideration

Certification agencies charge a high fee – ($200 per hour is perhaps an indicator and theappraisal period ranges from 2 days to 3 weeks). Suppose an appraiser rejectscertification to one of his clients, do you think that appraiser would get calls forappraising from any other organization? Therefore, the best that an appraiser would dois to cancel the assignment if he is dissatisfied with the preparation of the organization.Who would take the risk of being branded as too strict an appraiser? The organizationthat offers certification easily is the one most sought after.Another issue is that the certificate to issue certificates is being issued too easily. Passan exam and you get it or attend a training program and you get it. Go forth and multiplycertificates.Besides, these certifying organizations are profit-pursuing business organizations thathave expenses to meet, targets to achieve, and growth to be aimed at. That is thereason why we see a plethora of certificates being issued.

Method of appraisal

The appraisal process itself is under criticism – the appraiser looks at the evidencepresented to it. It is more like conformance audit – not an investigative audit. What is theguarantee that the evidence is not cooked up to suit the requirements of the appraiser?If financial accounting books (that are subject to statutory independent audit) are beingcooked, why can’t certification evidence be? I have had a call to cook the evidence for acertification audit and the organization head told me, with a straight face, that the