This action might not be possible to undo. Are you sure you want to continue?
Or: How to survive a career in software QA and still care about quality By Niall Lynch
I had not thought much about a career in software development until I arrived early one morning at the vocational school where I was teaching to find the body of the admissions director hanging from the ceiling fan in his office. There are certain moments in your life when you feel that the universe is not only trying to tell you something, but that it is shouting it at you through a bullhorn. That morning was one of those times for me. What could possibly be worse than this? I asked myself rhetorically, and immediately grabbed for the help wanted section of that day’s Chicago Sun-Times that the recently deceased admissions director had helpfully left on his desk. I did not know it at the time, but I was about to find out what could be worse than finding a dead body swinging from the rafters of your workplace: A career in software quality assurance. Which is not to say that software development doesn’t have its charms and perks, its good times and its moments of exhilaration. It’s just that software development taught me that a dead body can be taken away by the proper authorities, never again to reappear in your life. It is a problem, however ghastly, that can be solved. The problem of software quality assurance, the problem I was paid to solve day in and day out for 17 years, on the other hand, is one that has no such solution. Indeed, as I was to discover, it is a problem that only becomes more intractable, more terrifying the more successful you are at addressing it. Fortunately, on that cold and tragic morning, these insights lay far, far in my future, and all that beckoned to me then was the bright, glowing promise of a
Adventures In Quality Assurance - page 2 of 86
career in the squeaky clean world of software development, a world peopled by whiz kids who were, presumably, not yet suicidal. At least that’s what the ad in the Sun-Times told me. It seemed there was a software company that needed a manager of software quality assurance. I applied, interviewed, and within a month was newly installed as the Manager of Software Quality Assurance. That I was able to get such a job with zero experience in management, software development, or in quality assurance did strike me as odd at the time, but only odd in the sense of “lucky” or “charmed” or “someone up there likes me”. Little did I realize then that my ability to get a job in QA with minimal qualifications and experience was not a lucky break, but an omen. It was my first inkling that perhaps the software world was quite different from how I, and most of the people I knew, imagined it to be. The world of software is to this generation what the space program was to people in the 1960s. That is to say, a source of endless wonder, awe and optimism. Something shiny and ultramodern that defines how we think of ourselves and our world. A phenomenon that tells us that every year our world is going to get better, cooler, and more interesting for all involved. If you could go back in time to 1965 and tell people what was in store for the space program – the space shuttle crashes; the collapse into ineptitude and irrelevance; and, worst of all, the almost total indifference people would come to feel for the whole enterprise – no one would believe you. If you told them we would reach the moon first, but that, ultimately, it would seem like a pointless exercise, the people of 1965 would only get mad at you, and accuse you of being some kind of commie nut who was definitely “not with the program”.
By Niall Lynch – firstname.lastname@example.org 310-829-2044
Adventures In Quality Assurance - page 3 of 86
This book is, in a sense, the equivalent of that theoretical conversation back in 1965. Software books are invariably upbeat. They exude an aura of arcane authority, the promise of admission into secrets too deep for ordinary mortals. They relentlessly propagate the notion that the newest programs, the newest languages, the newest certifications are the new gateways to paradise, and that paradise is coded in Java. This is not that kind of book. I mention that now so that you will not be disappointed. As much as I would like to write such a book, I cannot. Moreover, it would be irresponsible for me to do so. Rather, this book is the fruit of many years of experience and reflection on the software industry as it actually is, on software as it is actually created. It is written from the point of view of quality assurance, but this is a perspective from which we can observe the decay of the software world in a nutshell, just as we can watch film of an exploding space shuttle and see in its catastrophe the state of our space program writ large. I write as someone who has seen it all, done it all, and has hundreds of T-shirts to prove it. I also write this book at a time when the unwavering luster of the software world is finally starting to fade. Software is no longer viewed as something out of science fiction that we can hardly believe we have in our hands. It is, rather, now seen as a commodity, something dull and uninteresting, no more capable of exciting our imaginations than the vacuum tubes in grandpa’s old radio. More than that, it has become a source of creeping anxiety. People now finally understand how great a risk we take as a society when we put more and more of our lives at the mercy of software. People are now starting to understand that all the technical wizardry and advancement that the software industry keeps
By Niall Lynch – email@example.com 310-829-2044
I offer this book as a kind of intervention. It is no more capable of seeing itself swinging from a ceiling fan than my friend the admissions director could see himself doing so in his youth. an attempt to prevent such a horrible outcome. I hope it succeeds. it is all too possible a fate. and that it is just as likely to work against us. This is not an accidental development.Adventures In Quality Assurance .com 310-829-2044 .page 4 of 86 trumpeting over and over again in its press releases doesn’t mean software will actually work for us. Optimism is being replaced by frustration. though this is something it cannot bring itself to acknowledge. Yet. It is in fact the inevitable outcome of how the software industry does its work. as I discovered that morning. By Niall Lynch – verlandosta@yahoo. and wonder by suspicion.
” With some minor adjustments. Not because those books are not often quite useful. For this reason I am not presenting yet another book on process.page 5 of 86 Introduction The British writer Horace Walpole once said. but it seems no one can do anything about it. Software quality is a bit like the weather. Rather.com 310-829-2044 . in fact. Any questioning of the status quo in re product quality is instantly transformed into questions of software process. So perfectly. because I have come to believe over the course of my 17 year career in software quality assurance that the obsession with process is being used as a kind of distraction from the deeper problems of software development. Nor because one can completely separate discussion of process from discussion of software quality. I assure you. because at the end of the day you can’t. and a tragedy for those who use it.Adventures In Quality Assurance . This is not an accident. which they are.” The tragedy comes from the chronically poor quality software that consumers have no choice but to purchase and use. Everyone likes to talk about it. co-opted and neutralized. and therefore cunningly misdirected into areas far from where the real problems lie. this aphorism can be made to express one of the fundamental truths about the software industry: “Software is a comedy to those who make it. famously. It is rather a reality to which the software industry is perfectly conformed. The comedy comes from how people in the software industry view product quality. but a comedy to those who think. that even the critique of this comedy from outside the industry has been completely anticipated. that “The world is a tragedy to those who feel. Everyone who works in product By Niall Lynch – verlandosta@yahoo.
even this purely technical By Niall Lynch – verlandosta@yahoo. but it is easier – and far safer politically – to pretend that process is the real problem. To this end. I believe that before I can offer you a better way of doing software quality assurance. such as the role of the stock market. The first part is an essay on institutional anthropology. In this respect. The second part of the book consists of a presentation of an alternative way of doing software quality assurance. its infernal persistence can never be fully understood. I must first expose and anatomize how the institutional realities of commercial software development conspire – knowingly. This essay will point many fingers. Because of this.Adventures In Quality Assurance . and ensures its persistence. and. there can be no real appreciation of how my proposals address what is really wrong with software quality assurance. willingly – to sabotage software quality in a systematic fashion. so fasten your seat belts – It’s going to be a bumpy ride. above all. the system of rewards and punishments that is actually in play in most software development environments. However. This is why the book I have written falls into two very distinct parts. software process has unwittingly become an enabler of these deeper problems. Absent this exposé. Unless one can place the problem of software quality firmly within the context of corporate politics and culture. I will encompass in my analysis factors that often are ignored in strict discussions of software process. the budgeting and scheduling process. an analysis of how the entire corporate culture of software development creates the problem of software quality.com 310-829-2044 . and is more what readers of books on software quality assurance are used to. and most also know how profoundly toxic they are to product quality.page 6 of 86 development knows what these deeper problems are.
because I believe such ideal proposals are doomed to failure. this book will not be doing the job for you. is a book about thinking. and probably never will. it is not something to be used as a kind of talisman to wave in front of questions that are inherently messy and difficult to make them disappear. Lastly. If you are willing to join me in this intellectual adventure. Consequently. I believe you have much to gain and nothing to lose. Though one hopes it is a little more well-mannered by the end of the second. For the idea of quality assurance I am presenting in the second half is itself a way of dealing with the realities I outline in the first. In short. I will not provide you with templates you can rip out of the book and distribute to your beleaguered QA staff. and follow its threads wherever they may lead. if you would rather have five bullet points you can e-mail to your managers. there are many other books that are better suited to meet that need. I will not be offering you five easy steps to best in class product quality. in its entirety. My book is not an excuse not to think deeply about the problem of product quality. this book. I suspect as well.page 7 of 86 section will rely upon and refer back to the analysis presented in the first part. Rather. However. It is not a purely ideal view of how to do software quality assurance.Adventures In Quality Assurance . I will be inviting you to think along with me. to unravel together the gigantic knot of product quality. And many other managers. The monster I describe in the first part has not disappeared in the second.com 310-829-2044 . bear in mind that they are meant to both deflect and co-exist with larger institutional realities over which quality assurance has no control. By Niall Lynch – verlandosta@yahoo. though my concrete proposals for how to understand. organize and execute quality assurance may at first seem alien or counterintuitive.
com 310-829-2044 .page 8 of 86 Let us wait a minute so that all those who cannot benefit from this book have closed it. There. placed it back on the empty business class seat next to them. Now the only ones left are people like us. Let us begin together. By Niall Lynch – verlandosta@yahoo. and gone on with their business.Adventures In Quality Assurance .
Part I : The Success of Failure .
poor quality software is one of life’s few certainties. holding forth some new process or. That most software developed in the world is of poor overall quality is well-known. is also well-known. Like death and taxes. He is always getting into accidents. that you have a very good friend who is a terrible driver. as the key to solving the problem. Of course. And so the cycle continues. software quality is not improved in any persistent fashion. even as other aspects of the software industry have matured and thrown off the growing pains of their youth. The relationship between the software process improvement movement and the software industry can be illustrated by a simple parable. Imagine as well that your friend realizes there is a problem.page 10 of 86 If It’s Broke. He is always getting citations for moving violations. Meanwhile. He sees the points on his license accumulating. it is also necessarily a book about the failure of software quality assurance. if you will.Adventures In Quality Assurance . These attacks generate a flurry of interest for a year or two and then dissipate. a phenomenon of such long standing in the industry that even to note its existence is to fall immediately into cliché. He sees his car insurance bill skyrocketing. It is a fact so well-known that it is no longer the cause of surprise or even of curiosity. As such.com 310-829-2044 . a new way of selling process. He knows that the cost of avoiding accidents and traffic violations is so very much By Niall Lynch – verlandosta@yahoo. software process theorists and the odd industry luminary mount attacks every now and then on the problem. Imagine. That this low level of quality has remained consistent over time. even better. Don’t Fix It This is a book about the nature of software quality assurance. to be overtaken in their turn by newer formulations of the same proposed solutions.
” your friend says. The engine is too weak to get me out of other driver’s way. Christopher statue to mount on the dash of his car. That the deficit is not one of engines and brakes and cup holders. So when he comes to you to discuss a solution. This makes you happy. One day he comes to you for advice about how to solve his problem. The brakes aren’t that strong. I was hoping you could give me some advice on what car I should buy to improve my driving. a little impatiently. There’s no navigation system. You buy an excellent instructional DVD for him on how to drive safely. I was thinking maybe a Porsche or a Ferrari. you make a list of driving schools you think will be perfect for him. solution-focused gifts. as you have seen both the danger his lack of driving skills poses to others and the constant anxiety it causes him. as his friend. so I have to hold my coffee with one hand while steering with the other. just to be sure all bases are covered. By Niall Lynch – verlandosta@yahoo. You even buy him a St. but the driver. The suspension doesn’t keep the car under control. You would also. which is why I’m always going the wrong way down one-way streets.” Your heart would certainly sink if you heard this from your friend. Every time he gets a ticket or drives into someone’s front yard. you are ready to give him your full support. Beforehand. he promises that he will do better. try to help him understand that the problem isn’t the car. and sweeps them aside. He is entirely committed to becoming a skilled driver. My problem is that I have a crappy car. “My problem isn’t that I don’t know how to drive.com 310-829-2044 . you happily present these aids to him.Adventures In Quality Assurance . And there are no cup holders. Imagine your surprise when he takes one look at your entirely practical. When he arrives. “Look.page 11 of 86 less than trying to fix them after the fact.
you will see hundreds of books on how to use popular software programs. and if you’re friend doesn’t understand that. How could it be otherwise? After all. by specialist magazines that tell him he needs a new car. If you walk into your local bookstore and browse the Software/Computing section. he will never get better. Since I am lazy and you are impatient.page 12 of 86 but of driving skill itself. Or do they? One of the main purposes of this book is to attempt to explain why this is so. the massive cost penalties of poor quality are well documented.Adventures In Quality Assurance . unlike in the parable above. You will see hundreds more books on popular computing languages.com 310-829-2044 . he is seduced by the simplicity of the solution. Our driver friend is the software industry in relation to product quality. Yet the phenomenon continues unabated. He is seduced by fancy advertisements. He doesn’t need to change. more powerful car. eerily resistant to all of the well-meaning attempts to eradicate it. It really is that simple. He doesn’t need to invest time in improving himself. Everybody knows this. All he needs to do is buy something off the shelf that will solve all his driving problems. Every theoretical failing of quality process and quality outcome can be traced back to that simple fact. Most of all. by the cachet that attaches to having a faster. few people in the industry seem to understand the fundamental mistake our friend is making. as is the endless agony of consumers who must discover and live with it. undisturbed. I will begin the book by telling you the answer: Software quality remains at consistently low levels because it is in almost everyone’s interest in the software industry for that situation to exist. The only difference is that. But your friend won’t listen to you. You will see still hundreds more books on how to prepare for certification tests in By Niall Lynch – verlandosta@yahoo.
a technical specialty on par with software engineering.com 310-829-2044 . It is also certainly of some relevance that the people who write all of this poor quality software are nevertheless lionized as wizards of high-tech. and the key career paths that exist within it. and generally are treated as among our society’s best and brightest. how could such a state of affairs be allowed to exist? And who could possibly benefit from it? It is certainly of some relevance to my point that many. many people in the software industry have become millionaires even though they continue to produce low quality product. But you will not see any books on software quality assurance. Why is this so? Again.page 13 of 86 hardware and software. nor even of marketing. You will find no guidebooks for how to ace your qualification test for Software Quality Assurance Tech Level IV – because no such certifications exist. If that conclusion is true. almost no attention is paid to the problem within the industry. despite its protestations to the contrary. if the software industry’s claim to be passionately concerned with quality were true. irrelevant to being successful in the software business. at the very least. and indeed does not exist as. What we learn from perusing such sections is that for all the rhetoric about the importance of software quality. By Niall Lynch – verlandosta@yahoo. Does this strike you as odd? It should.Adventures In Quality Assurance . For the composition of your average software book section faithfully reflects the priorities of the software industry itself. I submit that none of this could have happened unless low product quality was. must benefit from low product quality. then it must also follow that the industry as such. Quality Assurance is not treated as. as untouchable masters of an arcane art far beyond the ken of mortals.
which is admirable. Yet it is nevertheless the truth. to have their false assumptions exposed. But few if any attempt to extend their analysis further into the realms of corporate finance.com 310-829-2044 1 . That is why I am going to begin my discussion of software quality assurance by first analyzing the entire institutional context in which it operates. Socrates. also had keen insights into the marketing of software process consulting. that people are doing the wrong things because they are ignorant that they are wrong. as a whole.Adventures In Quality Assurance . Lacking this broader perspective. in fact. But this immediate narrowing of focus. one of the main ways that the software development industry preempts meaningful analysis of how it benefits from poor product quality. which is what most people do. and only one. Admittedly. It is. Which is to say. this reflexive direction of our attention to one. particularly if all you do is look at the quality assurance function itself. something that is rarely done. aspect of the software project process is itself a massive act of misdirection. By Niall Lynch – verlandosta@yahoo. process theorists can often identify what is being done wrong. This is why many software process improvement theories default to a Socratic view of the problem. it is a truth that can be difficult to discern. politics and subterfuge. this assumption Though of course it’s always easier to tell a paying customer that they are just misled by the ignorance of others instead of telling them they are the real problem. More sophisticated process theories will attempt to place the quality assurance function within the context of the product delivery organization. it seems. without being able to identify the real reason why it is being done wrong. at the level of the product delivery organization. Unfortunately. and its other functions.page 14 of 86 I realize this will sound like heresy to many. 1 All that needs to happen is for people to be taught the error of their ways. and they will want to do the right thing.
and how it can be made to succeed.com 310-829-2044 . To these questions we now turn. then it can only be…well. for all its manifest failings when judged from a theoretical perspective. the Standard System. Though it is one that is. we need to first identify the ways in which the Standard System. the workings of the Standard System will we be in a position to understand why software quality assurance fails.page 15 of 86 is itself entirely false. By this term I have in mind not only a description of how most software development organizations operate. but why they operate they way they do. Because if the cause isn’t ignorance. in detail. It is my contention that only after we understand. What I am going to do is to provide an analytical overview of what I will call. nevertheless generates massive benefits for those who participate in it. encouraged by the software industry itself. In particular. By Niall Lynch – verlandosta@yahoo. you see my point. perversely. for the sake of brevity.Adventures In Quality Assurance .
responsibility for the project’s remaining on schedule is transferred to another functional specialty. the sooner By Niall Lynch – firstname.lastname@example.org 16 of 86 I’m Failing As Fast As I Can Most software development projects follow a waterfall development model.com 310-829-2044 . The software coding phase is the responsibility of Engineering. most software project processes require the responsible function to achieve some specific milestone before they can transfer project responsibility to the group responsible for the next phase. And so forth. in the eyes of upper management. obvious what an enormous temptation this system injects into the project process. and therefore remove themselves from the hot-seat of blame. the “fault” for any project delays now lies with a different group. It is. most software development projects are structured so that specific phases of work are the responsibility of specific functional specialties. even if they think they are doing something different. even if all functional groups are responsible for some work within each phase. The requirements definition phase is normally the responsibility of Product Marketing or Product Management. Once that has been achieved. or should be. at least with respect to their scheduling milestones. The sooner Product Management can declare requirements “done”.Adventures In Quality Assurance . After that milestone has been met. Because this is so. at least. The testing phase is the responsibility of QA. That is to say. even if all the work they actually need to accomplish in that phase is not really completed. This system gives each functional specialty an enormous incentive to declare themselves “done” with their phase as quickly as possible.
They simply say. and there were no problems with them.” Note how this excuse carefully avoids the question of how those bugs got into the code to begin with. Or rather. bugs which customers promptly find. “If QA had found these bugs before the product shipped.com 310-829-2044 . It is the customer. it left many bugs undetected in the software. since it will just as surely be blamed for any project delays that occur during its phase. The searchlight shines only upon the last prisoner climbing out of the escape tunnel.” You can see where this is headed. but it rarely does. again. The beauty of this maladaptation is that it leaves most functions with a perfect way of deflecting blame. has no one to hand things off to. which is why QA generally takes the blame. Whereas QA. cause blame to rebound back onto the group that delivered the software.Adventures In Quality Assurance . the pressure on QA to declare itself done is just as great as for any other function. is the beauty of the system. we would have fixed them. Nevertheless. You would think that this would.page 17 of 86 they can blame Engineering for any project delays. which. except the customer. This allows Engineering to say. By Niall Lynch – verlandosta@yahoo. not upon all those who fled before. ultimately. so blame must lie with the next group in the chain. it rarely does effectively. the sooner they can blame QA for any project delays. then. Having declared itself done prematurely. who inherits the often disastrous results of this institutional adaptation. being the caboose on this bullet train of shifting responsibilities. “I got all my deliverables done on time. The sooner Engineering can declare “code complete”. The last link in the chain is the one with the least ability to offer a plausible excuse.
This explains why software projects always seem to go so well at the beginning. If this sounds like a very perverse bargain. There are few. It creates a process where there are no meaningful exit or entrance criteria for any of the early project phases. the Standard System optimizes for political risk mitigation over project risk mitigation. In essence. since such mitigation would force each function to spend more time validating its work. the Standard System undermines rational risk mitigation. there is nothing stopping people from just beginning. and then jump off the rails in the final stretch. There are two fundamental truths that emerge from the analysis above. moving it all to the end of the project.page 18 of 86 This basic orientation of the Standard System has one very important overall effect on the software development process. By Niall Lynch – verlandosta@yahoo. but also why knowledge of delay comes only at the very last minute. welcome to the world of commercial software development. truths which must be openly acknowledged before any group can understand its quality failings. This explains not only why project delays often come as a complete surprise. in any phase of the project. the Standard System habitually end-loads project risk. true mechanisms of validation in place that must be satisfied before work can begin on any function of a project.com 310-829-2044 . if any. and thus assume a much greater risk of being held accountable for schedule delay during their phase. That is to say. QA just starts running tests. In this fashion. Product Managers just sit down and start writing requirements. To put the point in a different way.Adventures In Quality Assurance . Engineers just start writing code.
for the process described above to work. People don’t get degrees in SQA. and CYA-effectiveness. for QA. and the best SQA staff usually try to leave it as quickly as possible to secure a job in engineering or product management. These truths help us understand one otherwise puzzling aspect of software development culture: the persistent weakness of SQA. and sometimes not professionalized at all. for so long. Second. Because if it doesn’t. therefore. a powerful department or group within a software development organization.Adventures In Quality Assurance . of course. the process described above is only a maladaptation from the point of view of abstract process theory. From the point of view of internal politics. QA must fail. it is a necessary output of the system. the sine qua non of its surreptitious success. even though product quality is acknowledged by everyone as the key to customer satisfaction and. customer dollars? It’s really not that big of a mystery. Salaries for SQA staff and management often don’t begin to match the salaries paid to engineers and product managers. SQA is rarely. SQA is universally viewed as a lower level function. It is not a result that persists because of people’s ignorance of its causes. One only has to note that SQA’s very role is defined as that of catching the mistakes of everyone further up the project By Niall Lynch – verlandosta@yahoo. this process is a positive adaptation that delivers consistently positive results. So the persistent failure of QA is not an accident. Let me say that again: QA must fail. SQA remains only half-professionalized. How could this situation exist. Except. career enhancement. if ever. Rather. then it cannot provide cover for failings further up the product delivery food chain. and persist. Though commercial software development is at least four decades old. even though it is a vital project function.page 19 of 86 First.com 310-829-2044 .
on Product Management. exposing major shortfalls of fundamental expertise in groups far more powerful than QA. failings for which only Engineering could take the blame. They are there to inherit the results of all the tasks that have been left undone earlier in the project.com 310-829-2044 . That cannot be allowed to happen. to try to “fix” them in the QA phase and thereby take responsibility for them when they are not. 2 I trust by this point the quotation marks around this word are self-explanatory. If QA functioned the way everyone claims they want it to. then the question. systematic failings of the product architecture. This cannot be a coincidence. Moreover.page 20 of 86 food chain. Its findings would necessitate massive rewriting of the code. “Why is QA not professionalized?” answers itself. which means they cannot be trusted with the kind of authority and autonomy granted other project functions.Adventures In Quality Assurance . necessitating a delay that could only be blamed upstream. Especially when we realize that a professionalized SQA function would wreck the system. it would uncover. which in turn would lead to significant schedule slip. If we understand this. the continual “failure” 2 of QA only gives other project functions a greater justification for dominating and controlling it. After all. contradictory or inadequate requirements late in the process. It would uncover massive. the consequences of which could not be laid at QA’s door. for example. A professionalized QA would shine the spotlight upstream in the development process. The circle of life is complete. they are not doing their job correctly. By Niall Lynch – verlandosta@yahoo.
Some good QA people stay on. usually for personal reasons. The second question that arises is: Why on earth would executive management put up with such a system? After all. they never have to take true responsibility. but the other departments are too dependent on a non-functioning QA group to allow that to go too far. a QA staff that is carefully selected by the system to thrive on its injustices. and cost the company money and customer goodwill. not exactly.Adventures In Quality Assurance . they get beaten up every now and then when a product blows up in the field.com 310-829-2044 . the endless streams of problems in the field produced by this system do nothing for their personal PR. By Niall Lynch – verlandosta@yahoo. Moreover. The first is: What’s in it for QA? Why would QA put up with such a system. The result is. generally. least welltrained QA staff in place. Since they are never given true autonomy. and to reinforce them for its own benefit. cause endless headaches for them.page 21 of 86 Two questions naturally arise at this point. As I pointed out above. QA itself can point to its relative lack of authority as an all-purpose excuse. where they take all the blame for everyone else’s mistakes? The answer is simply that good QA people usually don’t. As with the other levels of the system. each failure of QA leads to a diminishment in what is expected of QA. all is not quite as it seems at the executive level. Sure. a situation which plays into the hands of those QA staff who do not want the responsibility and true accountability that would come with real power. And for such people there is a perverse kind of protection in the system. Right? Well. but normally the QA function winds up with only the least committed. They leave QA as quickly as possible once they realize it’s a game they cannot win.
like a shaken can of soda. and the executive level. Most commercial software companies are not run by people with engineering backgrounds. The winnowing process that selects executive staff tends to weed out engineers. and support the software. failures and questions about C compilers. write. because the engineers are always reminding them (with all the tact and understatement for which engineers are justly famous) how ignorant they are of what the engineers know and do. 3 This creates an executive culture that is mystified by the engineering process.Adventures In Quality Assurance .. test. Neither is an appetizing prospect for most executive staff. it winds up spraying them with problems. executive management generally want to know nothing of the details of product delivery. and their respective management hierarchies).page 22 of 86 The first step in answering this question lies in understanding that there is usually a huge cultural gap between the product delivery level of most software companies (i. and not a little intimidated by it. As a result. and that of being tarred with its many failures. There is often a thinly-veiled hostility between the executive level and the engineers they pay so well.com 310-829-2044 . As a result. executive staff often regard the product delivery process as a black box that they are afraid to open. and favors people with backgrounds in sales or finance. the groups that actually specify. lest. These cultural peculiarities create a strong incentive for executive staff to let their product delivery groups run their own show with as little oversight as 3 Engineers usually lack the communication skills and business savvy to be compelling candidates for executive positions. That’s what they pay the engineers to take care of. To interest themselves too closely in its functioning places them in a double peril: that of clashing with the engineering staff. By Niall Lynch – email@example.com.
the very same product delivery staffs that caused the problem in the first place. by themselves. The important thing to understand is that this impulse only becomes stronger. In other words. 4 Though it needs to be pointed out that this does not prevent them from taking credit for it after the fact. that is a miracle that they. the apparent failings of the Standard System produce a series of stellar opportunities for conspicuous self-sacrifice and heroism within the product delivery group. blame and pink slips in the name of solidarity. often for months on end. These in turn surround product delivery with a halo so bright. disaster is a radioactive substance. that afterwards it seems petty and ungrateful to probe too deeply into why the crisis happened in the first place. to be approached only through layer upon layer of the proper shielding. which. we arrive here at one of the darkest truths of the Standard System: It is designed to produce such disasters. and they know it. usually means subordinates.com 310-829-2044 . as quality disasters multiply in the field. So crisis time tends to be the time when everyone pulls together. working mega-overtime. Indeed. In corporate life.page 23 of 86 possible. Bizarre as it may sound. By Niall Lynch – verlandosta@yahoo. eschewing finger-pointing.Adventures In Quality Assurance . The last thing you want to do is grasp it in your hand. Unfortunately. Preferably from groups other than one’s own. because they only enhance the power of those who are responsible for them. in the corporate world. executive management only want it to go away as quickly as possible. 4 Only the product delivery staffs can get the bad PR monkey off their backs . it is at the moment of greatest disaster that engineering groups generally have the most power over executive staff. are incapable of working. not weaker. When a quality meltdown occurs in the field. Crisis time is also when engineering and QA whip themselves to new heights of selfless devotion.
ultimately. a crisis of new product development instantly takes its place. Failure to meet announced product delivery dates will only lead to uncomfortable questioning from large customers and the investment community. that you will consider your most trusted comrades. these periods of gladiatorial exertion on behalf of the company are major bonding events within the product delivery group itself. crises that eat up massive amounts of precious new product development time. It is not at all unusual. They become something very like a narcotic for software people. however much they may complain about them afterwards. and most fondly. Indeed. and. what prevents executive management from punishing product delivery for its failings after the crisis is past? This question unveils the final piece of diabolical brilliance inherent in the Standard System. it is often the people whose negligence contributed most directly to the disaster who are most greatly honored. This phenomenon is so consistent that it is difficult to avoid the conclusion that product delivery groups are happy to provoke crises in order to create opportunities to be noticed and rewarded. the heroism of product delivery during these crisis periods often generates concrete rewards for them. Cheetos and superhuman stamina. for product delivery staff to be given bonuses and public approbation in the form of awards. in their later years. It is these crisis events that software people tend to remember most intensely. As incredible as it may seem.com 310-829-2044 . Nevertheless.page 24 of 86 Moreover. the threat By Niall Lynch – verlandosta@yahoo. The system continually generates crises in the field. sustained only by coffee. So once an in-field crisis is past. once the smoke has cleared and the stock price has recovered its value. It’s the people you’ve worked with till six in the morning.Adventures In Quality Assurance .
page 25 of 86 of a lower stock valuation. so that any attempts to replace it will cause the company to implode. And the only way they can get through the month is to pile more debt onto the load they already By Niall Lynch – verlandosta@yahoo. To do so would only push the company further behind. further into peril. The software development system has immunized itself to such retribution. This is the reason product development groups can fail again and again. racing to put out the next fire. They would love to save money. of course. because everyone must keep racing. no time to exhaustively revamp product development. The system leaves no time for such things.Adventures In Quality Assurance . such an irresistible momentum into its own future. that it is almost impossible to interrupt it long enough to change it. The system has booby-trapped itself. be missed in their turn. make the current quarter’s numbers. These dates will. racing. precisely by being so inefficient that it cannot be interrupted long enough to correct its structural “failings” without further risking valuable product delivery dates. There is no time to mount inquisitions. meet the next delivery date.com 310-829-2044 . but they can’t because otherwise they can’t make their debt payments. The strategic leveraging of perpetual crisis is also the way that many software development groups immunize themselves against process improvements. In this respect. least of all executive management. The system creates and sustains such a compelling urgency. the software industry is very much like someone saddled with huge amounts of consumer debt. and yet escape any kind of significant retribution from executive management. but no one wants to admit that at the beginning. no time to reengineer the engineering department.
The only weapon executive management really has in its arsenal is the purchase of a new product delivery group through the acquisition of another software company. Nevertheless. and crowning irony. Executive management winds up purchasing another development group that is every bit as committed to the system we have described. This need leads in turn to the new development group being given free rein to reinstitute.page 26 of 86 have. and so is to be avoided at all costs. It is a self-perpetuating loop that quite effectively shuts off all the obvious escape routes. It is in fact the desperate need to cast off perpetually failing development groups that drives many software acquisitions. only to wind up marrying a drug addict. The remote. At this point in the drama.com 310-829-2044 . can never be openly acknowledged.Adventures In Quality Assurance . Many software engineering groups are like a credit card that can never be paid off. executives have to demonstrate that they are exercising some form of effective control over the projects that fall within By Niall Lynch – verlandosta@yahoo. and sometimes even improve upon. especially after spending countless of millions of dollars on something they thought would solve their problem. of course. As we have seen. Executive management is like the wife who divorces her alcoholic husband. though this. the dysfunctional functionality of the Standard System. hands-off approach that executive management takes to product development has another manifestation that is a key contributor to the Standard System. . a group that is every bit as adept at all the strategies we have already analyzed. Yet executive management desperately wants to believe otherwise. diving too deeply into the details of product development erodes an executive’s all-important plausible deniability. we encounter the final. yet also can never be maxed out.
since project milestones are seldom direct measurements of concrete progress. they may not want to know anything about whether they are getting done. I have seen projects that shipped only a few weeks beyond their By Niall Lynch – verlandosta@yahoo. since project managers well know that conformity to schedule is the single metric they will be judged by in the eyes of executive management. In my own career I have seen this principle at work. I have seen projects that shipped within schedule.Adventures In Quality Assurance . Schedule management is the one thing they are quite keen to know a lot about. The solution that many executives seize upon is schedule oversight. where the goal of each functional group is to declare themselves “done” as quickly as possible. most product delivery managers learn very quickly how to manage projects so that they always appear to be “on time”. even if in reality they are not progressing in any substantive sense. Indeed. insulates executive management from having to get involved in the details of any developing disaster. of the phenomenon described at the beginning of this section. yet which then produced major quality and customer satisfaction disasters in the field. I have put both those terms in quotation marks. in turn. they are solely measurements of conformity to schedule. whereas projects that are “late” make them look bad.page 27 of 86 their portfolios. so that blame for any schedule nonconformances will fall on the next group in the chain. They may not want to know anything about how things are getting done. Their essentially self-referential quality is what. Rather.com 310-829-2044 . but they have an obsessive interest in when project milestones are met or not met. singled out internally as “success stories”. the true stimulus. This is the real foundation. since projects that are “on time” make them look good.
By “dollars” I mean the amount of budget they control and the size of the product revenue streams that are accounted to their group. By Niall Lynch – verlandosta@yahoo. And. Logically. and they in turn have internalized it so completely that it operates as an institutional reflex. the antidote to disbelief is to understand how the system rewards conformance to its true priorities. have the largest budgets. When executives who are all formally at the same level.com 310-829-2044 . since that is not what their own success is going to be judged on. and are accounted the largest revenue streams in the company. By “bodies” I mean the number of staff they control. like every other function. This prioritization is communicated with crystal clarity to the troops. then. In this fashion. have no real stake in product quality. with the same title.page 28 of 86 committed ship date.Adventures In Quality Assurance . want to compete with one another. executives are going to manage their groups and their own careers so that they wind up with as large a share of all of those pies as they can wrest from the control of their colleagues. The most powerful executives are those who control the largest numbers of employees. nevertheless tagged internally as “failures” because they did not meet their original ship date. They. executive management reinforces every aspect of the Standard System. you may find this a surprising reality. Again. You need first to understand what constitutes real power in a corporate hierarchy. again. something that does not require discussion or comparison with other optimizations. they battle over two things: bodies and dollars. which generated high in-field quality and customer satisfaction results.
Executive management is already looking for any reason to move your resources and budget dollars over to new. even though they have neither at the moment. but are rather publicly acceptable formulas for expressing the fact that executive staff are bored and need new distractions to keep them interested in the business. Often it is projects that promise to create whole new markets.com 310-829-2044 . the products that generate the most revenue are considered “legacy projects”. Particularly in a publicly-held company. One of the most maddening paradoxes of the product development system is that success in the marketplace often leads to your project being disregarded in this fashion. but this is not always true. it should not be difficult to guess what kinds of projects are likely to be rewarded with more bodies and more budget dollars: Those that demonstrate schedule compliance. on which they have pinned all 5 These judgments on the part of executive management rarely have any basis in fact. By Niall Lynch – verlandosta@yahoo. Projects that have faithfully met their schedule milestones are the ones that will be the best candidates for receiving more budget dollars and more staff. 5 Therefore. and whole new revenue streams. ‘sexier’ projects. If we analyze this paradox further.page 29 of 86 Given the corporate priorities I have outlined. In many cases. and thus unlikely to generate significant new revenue. serving markets that are already “saturated”. You may think that those products which generate the largest amount of revenue would automatically receive the largest budgets and staffs. they are not seen as good candidates for higher levels of investment. we see that if you are managing a cash-cow “legacy” project. that are prioritized for new investment. regardless of what other effects may follow. shipping “on time” is the most crucial factor.Adventures In Quality Assurance . efficiency is not necessarily in your best interests.
otherwise. on the assumption that you must have some slack in your organization if you are delivering product without drama or fanfare. or rather by what would seem to be By Niall Lynch – verlandosta@yahoo. this strategy does nothing for product quality either. you will never receive any more. since schedule crisis almost always leads to quality crisis. This is why project managers learn that it is not necessarily a bad thing to cause a crisis every now and then. So if you consistently demonstrate that you can deliver new versions of your legacy project with the staff and budget you already have. that the standard system is fueled. that you need more. the best way to attract more money and bodies to your project is to get into trouble. This is a particularly effective strategy if some huge percentage of the company’s quarterly revenues come from your product. Because then the company has to give you whatever you need. I have tried to demonstrate two realities in this extended analysis of the Standard System. More than likely. an executive VP will instruct you to “do more with less”. that you can’t be taken for granted. how would you be able to run your project so smoothly? On the other hand. at every level. from top to bottom. otherwise they face stock price meltdown. quiet efficiency routinely makes upper management suspect that you have more than you need. Unfortunately.page 30 of 86 their hopes for new sources of revenue. So the truth of schedule compliance turns out to be a little more complex than it first appeared.com 310-829-2044 . Strange as it may seem to outsiders. yet too much schedule compliance makes you a candidate for budget and resource cuts.Adventures In Quality Assurance . as a way of demonstrating that you don’t have enough. by a will to dysfunction. Schedule compliance is the sine qua non of success in the corporate system. First.
But my second point is that. it is eminently repeatable. than many of the more elaborate processes that are proposed as its replacement. Secondly. in fact. Far more repeatable. and can articulate its basic features without much prompting. it is lightweight and. yet relatively few of them have seen. It does not add layer upon layer of bureaucracy to monitor its effectiveness. it is supremely functional. the system is identical with its rewards. it has a leg up on certain other proposed software processes. but also a way of consistently rewarding those who comply with it.page 31 of 86 dysfunction and folly to outsiders. On the contrary. The Standard System can adapt to almost any new circumstance. In this respect. not something apart from it. it is not simply a way of doing things consistently. For one thing. the second. and still maintain its essential characteristics. Most importantly of all. Not so the Standard System. It does not require extensive and expensive training. quite agile. many software process theorists don’t want to acknowledge that the Standard System already possesses many of the hallmarks of good process. the system is not dysfunctional at all. or have allowed themselves to see. Or to state the point in a more ironic way. Everyone who participates in it knows what it is. looked at purely from an internal perspective. whose benefits are often abstract at best and require quite a long time to materialize. The process is how the company does business. By Niall Lynch – firstname.lastname@example.org 310-829-2044 . Many process theorists have seen my first point quite clearly. It is a grand optimization for success that delivers the goods time and time again. That is to say. It also complies with one of the key requirements of many more theoretical processes: it is both a software process and a business process. in its own perverse way.Adventures In Quality Assurance .
If we understand this. in spite of all its manifest and consistent failings in many areas (such as product quality). Of course. They declare the experiment a “success”. by contrast. ceases to be any kind of enigma. Yet many powerful forces conspire to prevent the software industry from evolving. “Process Improvement” is a religion that certain management structures convert to periodically. it is worthwhile to focus more deeply on the crucial role software engineering plays in maintaining it. and on that basis instantly revert to their original way of doing things. Rather. Even people enmeshed in the depths of the Standard System realize this. The most powerful of which is software engineering itself. They are like the person who gets drunk the minute they get out of rehab. there are reasons why one might want to do things differently. I would argue. to celebrate the fact that they’ve been cured of alcoholism. the persistence of the Standard System.Adventures In Quality Assurance . the relevant question becomes instead.com 310-829-2044 . dispensing its rewards instantaneously. Before we finish our analysis of the Standard System.page 32 of 86 It is. only to abandon it after one or two iterations. a veritable Dr. By Niall Lynch – verlandosta@yahoo. that is both unique and uniquely powerful. a role. “Why would anyone want to replace it?” Well. Feelgood. the honest answer is: Almost no one does.
in detail. We cannot get to the root of why QA remains unprofessionalized unless we first examine. The first step in this inquiry is to point out how software development differs from most other professions.com 310-829-2044 . As anyone who has worked in software development can attest. architects and other high-level professions. Yet it would be equally naïve to believe that this position of centrality does not produce its own unique distorting effects on the system as a whole. 6 there is nevertheless a huge difference in how software engineers come to their careers. members cannot do much of anything until they are first 6 This popular association is due exclusively to perceived economic parity between software engineers and the traditional professions.page 33 of 86 Lords of the Manor So far we have talked about the software development system as a whole. Only a very naïve person would find this preeminence surprising or unfair. It is the engineers who write the code which forms the product that makes it possible for the company to earn its revenues. But this is not really true. the engineering group is the sun around which all the other functions orbit. Judges don’t cut their teeth by first trying a couple of dozen cases before going to law school. To be fair. judges. and especially in the case of QA. In all real professions.Adventures In Quality Assurance . attributing to each of its components a fairly equal share of responsibility and benefit. this makes perfect sense. the role Engineering plays in this phenomenon. and only then going to medical school. Though in the popular imagination software engineers are classed with doctors. Surgeons don’t begin their careers by first operating on people for years. By Niall Lynch – verlandosta@yahoo.
com 310-829-2044 . when they have written lots of software already.page 34 of 86 certified by a professional body. purely on the strength of their demonstrated skill at writing code. in fact. still willing and able to recognize talent. This is why so often they treat everyone else as a superfluous ninny who is only getting in their way. there is a strong consciousness among software developers of not actually needing any of the other functions inherent in commercial software development. irrespective of the paper it comes wrapped in. More importantly. As a result. However. before writing software. from a non-engineer to boot. Most of them began coding as teenagers.Adventures In Quality Assurance . In other words. Software engineers are nothing like this. when they have produced lots of software already without requirements or product managers? Why should they submit their code for testing. all by themselves. To this day. in real professions mastery follows professionalization. without having some QA group first having to approve it? By Niall Lynch – verlandosta@yahoo. one of the cool things about the software industry. used by lots of people. It is still porous. This is not a bad thing. long before ever entering an electrical engineering program in college. without the help of any other people or functions. and many have produced functioning software. many software companies will hire a software engineer who has no formal degree in the subject. Why do they need formal requirements. this is an expertise they believe they have acquired and demonstrated all by themselves. this career path also means that many software engineers feel complete in their understanding of their discipline before they ever get their first formal job in a software development group. It is. after many years of preparation and training.
for example. Why. who produce and sell their first product without any help from any other functions. Software engineers as individuals. Each engineer writes in their own dialect. it can only be defined in relation to the needs of software developers themselves. they find it so difficult to cooperate even with other software engineers. By Niall Lynch – verlandosta@yahoo. and the software industry continues to reap the consequences of this inescapable fact. 7 This profound sense on the part of software engineers of the superfluity of all other software development functions means. This usually leads to high drama worthy of an extended episode of Dynasty.com 310-829-2044 . The path to professionalization in software development is fundamentally solitary. This fact explains a lot of things about software engineers. that they find it impossible to attribute real autonomy to them. Consequently. Why.Adventures In Quality Assurance . any real use. and it’s not hard to understand why software engineers find it difficult to comprehend why anyone else should muscle in on “their” territory. where programmers are actually forced to cooperate with each other. Software engineers often believe they have acquired and proven their expertise prior to any experience of professionalization or socialization into a larger business environment. which is largely incomprehensible to any other engineer. and software engineering as an institutional entity within a corporation.page 35 of 86 Add to this the fact that many software companies are founded by a sole engineer. to this day software engineers cannot understand each other’s code. for example. or very small group of engineers. habitually feel it is their right to define the duties of 7 If you doubt this. institute the XP (Extreme Programming) process in your organization. One of its key features is “pair programming”. If they can be given any real purpose. software engineering has a strong tendency to view other software development functions as their servants. in turn.
In institutional terms. such as Product Management. As a result.com 310-829-2044 . QA staff are often chosen by Engineering managers. departmental infrastructure. Moreover. for example. This is sometimes true of Product Management. according to criteria they deem sufficient. QA does not even exist as a separate department.page 36 of 86 other functions. Often attempts to develop these other interfaces arouse the immediate wrath of Engineering. or budget. because they have very strong “uplinks” into sales and/or finance. It is not uncommon for QA to lack its own management hierarchy. Engineering’s domination of QA means that QA is never really institutionalized in the first place. not to the head of the business unit. In many organizations. and report up through the project hierarchy. QA staff are distributed to each project. who sees them as a form of betrayal. it is the most completely colonized of all the other software functions.Adventures In Quality Assurance . QA is normally completely dominated by Engineering. However. Some functions are able to resist this colonization to a degree. Engineering’s domination of QA means that QA’s methods and duties are not defined in terms of the needs of QA itself. an attempt on QA’s part to “engineer” its own independence. though QA almost never has similar veto power over engineering staff. QA never develops mature interfaces and relationships with any other project functions that are equally necessary to product quality. It exists only as a para-function. In functional terms. lacking its own hierarchy. that mirrors that of Engineering at every level. QA does not have this advantage. QA usually reports to the head of Engineering. and to exclude from that definition anything that does not suit their immediate needs. Rather. nor in By Niall Lynch – verlandosta@yahoo. And on and on.
In professional terms. but rather solely in terms of what Engineering needs QA to be and to do. This leads in turn to a situation where Engineers offload onto QA staff many tasks and responsibilities that logically should rest with engineers themselves. It enforces upon QA a definition of its work that is completely task oriented. is reduced to a group of people who do nothing but push buttons and find bugs (one hopes). without being able to relate that data to any meaningful concept of product quality in general. This is why. In philosophical terms. This is why.page 37 of 86 terms of what the company may require in terms of quality. QA is nothing more than Engineering’s valet. nor even in terms of the specific product under test. for example. Hence the career path of many of the most talented people in QA is…straight out of QA. This is why QA. They realize early on that QA is not a player. It can only do what Engineering wants it to do. this domination means that QA staff really have no career path. and as quickly as possible. many engineers feel they need QA staff to immediately test every line of code they write to “make sure it works”. not goal oriented. in many software organizations. the “release criteria” of most software projects consist solely of bug metrics (“No open Category A issues. this domination ensures that QA never develops a clear idea of its own discipline.com 310-829-2044 . No By Niall Lynch – verlandosta@yahoo. In many software organizations.Adventures In Quality Assurance . and no more. and that they themselves will not become players if they stay in it. for example. Why the engineer should not be primarily responsible for this is a question that is rarely asked.
I busied myself with being sure the QA staff had a good test plan. I was assigned a specific project that my boss wanted me to oversee personally. soon after becoming Director of QA for a medium-sized company. and we worked out a very realistic project schedule that everyone was happy with.Adventures In Quality Assurance . etc.com 310-829-2044 . even though bug metrics.” etc. while minimizing its own accountability to a vast degree. That is the area of accountability for the quality of the released product. By Niall Lynch – verlandosta@yahoo. Meanwhile Engineering reserves to itself complete authority. We finished the requirements phase and entered the coding phase. These claims are easy to illustrate from my own experience. He had trouble understanding my point. Everything was going fine. The Engineering Manager told me that his group would be six weeks late in achieving the code complete milestone.). For example.page 38 of 86 more than three open Category B issues. I met with the Engineering Manager and his team. “Why will it 8 If you doubt this claim. 8 There is one area. by themselves. and vehemently resists any attempt to wrest control of QA out of their grip? After all. tell you next to nothing about product quality. only to immediately manifest nothing but “Category A” issues in the field. simply recall to mind the number of software products that are released with “zero open Category A issues”. I told him we needed to notify management immediately that the project was going to be a month and a half late. until the week when Engineering was to declare itself “code complete”. however. Though QA is denied any meaningful authority. that Engineering often feels fiercely possessive of “its” QA staff. where Engineering prefers to let QA take the driver’s seat. Is it any wonder. it is nevertheless required to assume complete accountability. that tests were being written. given this convenient arrangement. good help is hard to find. that relevant equipment was being obtained.
because of QA. while QA’s was infinitely By Niall Lynch – verlandosta@yahoo. There is only one explanation for this attitude: Engineering had habitually. Fortunately. but clearly bewildered.Adventures In Quality Assurance . hence the release date would have to be moved forward by the same amount as the delay in reaching code complete. informing them that the project was going to be late. He immediately shot off an e-mail to upper management. Engineering clearly thought that its own schedule was infinitely expandable. I pointed out to him that the delay in reaching code complete was a delay in when we could begin testing. upper management accepted my facts and conclusions. pointing out that it was in fact Engineering that was going to be six weeks late. been able to blame QA for its own inability to get its work done on time. He was not engaging in political maneuvering. It was as though he just couldn’t understand what was happening. “Yes. It simply never occurred to him that Engineering would have to be accountable for its inability to meet its milestone commitments within schedule.page 39 of 86 be late?” he said. over a long period of time.” Needless to say. and that QA had not lengthened its own time estimates for the testing phase by a single day. I had no choice but to send a clarifying response. “So you’re saying that QA can’t make the date?” he replied. he blew a fuse.com 310-829-2044 . This is important. The Engineering Manager in this case was genuinely convinced that it was QA that was making the project late. The Engineering Manager was not only livid at this result. that’s what I’m saying. not Engineering. Because Engineering has not made its dates. There is another important detail in this story.
Engineering could assume a six week extension of its own schedule. Most definitions of product quality are not. in fact. wishes that attach themselves to nothing truly knowable or measurable. it could expect QA to lop six weeks of its schedule because there is no real operational definition of product quality. no major ones that the reviewers will find. such that one could unambiguously describe the effect on the quality outcome of those missing six weeks. on the contrary. This is in turn an artifact of defining QA in terms of its tasks. It could do this because Engineering had a very specific and complete understanding of its own discipline. no major bugs the reviewers find that we can’t quickly fix and send out as a patch…” And so forth. Or at least. it is really. This can be By Niall Lynch – email@example.com 40 of 86 compressible. Or at least.com 310-829-2044 . Though many people believe that QA has a well-defined mission and goal. Conversely. simple expressions of desire. and fundamentally. an assumption about quality itself. They are. Or at least. definitions at all. it generally does not. but what the results of completing those tasks would be. The unquestioning assumption on Engineering’s part was that QA could cut six weeks from its test schedule without affecting product quality in any way. no major ones. This claim may seem obviously false to many of my readers. rather than its goals. It could list not just what tasks would be accomplished. Though this seems at first to be an assumption about QA. because it could specify what exactly it would accomplish in that six weeks’ time. Yet Engineering would never have made that assumption about its own work. “What do you mean there’s no definition of product quality? There is in fact a very clear one! No bugs. They are probably thinking at this point.Adventures In Quality Assurance .
my first project was to organize the testing of the installation programs for the company’s first multi-tier (i. I waited for my response.Adventures In Quality Assurance . detailing all the files that are installed for each installation type. Within minutes of arriving on my first day. one normally will find only vague stipulations. Yet these metrics themselves are fools gold – they are not attached to any meaningful definition of test coverage nor. it’s OK to laugh. quantify nothing but the illusions they are meant to support. And waited. and where they are installed for each installation type. I answered by explaining to him that no testing had occurred. and so.” 9 He ended his reply by demanding to know how far I had progressed in testing installation that day. “clientserver”) product. I received an e-mail from the VP of engineering. Finally. In the section where QA must define its quality goals. 9 Go ahead. The clock on my desk ticked off valuable hours. where I was also the QA Director. by themselves. to any meaningful validation of product requirements.” Often there is an attempt to pseudo-objectify the definition of quality.page 41 of 86 easily confirmed by perusing almost any QA project plan. No problem. because. By Niall Lynch – verlandosta@yahoo. by expressing it in terms of bug metrics. who informed me that no such specification existed.” or “Zero major defects. I wrote back. at the end of the day.com 310-829-2044 .e. and that engineering was not going to produce one because that would “slow the project down. At another company. Perhaps another anecdote will illustrate this point more clearly. and which directories and subdirectories are created as well. like. “Best in class quality. I got a response from the VP. All I need from engineering is the installation specification. more importantly. stating that this was my top priority and I needed to have it done ASAP..
This time the VP required only a few minutes to reply.page 42 of 86 logically and operationally. it illustrates the extreme hostility QA will encounter from Engineering By Niall Lynch – verlandosta@yahoo. Second.e. Fourth. First. I had to escalate the problem to our mutual boss. Go fuck yourself. it illustrates how Engineering has no concept of product quality. who immediately saw the logic of my request. It was brief and to the point. Such a reply is far from uncommon. Needless to say.Adventures In Quality Assurance .com 310-829-2044 . the VP of Engineering became my sworn enemy from that day forward. How would we know if it was failing to install necessary files? Etc. it illustrates how Engineering views QA as a group whose job it is to do only what they tell it to do. This story illustrates in a pithy nutshell all the key institutional realities I have described above. Third. I sent the reply off. it is impossible to test a product’s subsystem when you have no idea what it is supposed to do.. even from VPs who should know better than to express such sentiments in writing. so that they can offload responsibility for schedule slip onto QA as quickly as possible). convinced my logic would convince him. since there is no other way they could think the “quality” of an installation program could be “assured” without QA having a clue regarding how it was supposed to operate. Our boss was also unpleasantly surprised that no specification for installation existed. How would we know if it was installing the wrong files. Shocked? Don’t be. it illustrates how engineering often offloads onto QA core responsibilities of the engineering function itself – such as determining what installation is actually supposed to do – under the guise of “moving things along” (i. it said. to the wrong locations? I asked.
and its own demands on Engineering. Logically. A By Niall Lynch – verlandosta@yahoo. with its own goals. No one finds anything odd or contradictory about this name. that it has been able to perpetrate an amazing feat of institutional ventriloquism. and taking responsibility for all of Engineering’s failings. Engineering should be held formally responsible for “quality assurance”. and so unthinkingly accepted by other project functions. Hey. Which is quite surprising actually. The quality of software is a function of the quality of the code that comprises it. Because if it did come into existence. Engineering’s domination of QA is so complete. when it is obvious that QA does not “assure” the quality of anything. Does “Quality Assurance” write the product code? No. a ventriloquist’s dummy operated by Engineering. then what the heck is it? Excellent question. and also to generate a list of deliverables and responsibilities that Engineering owed to QA.com 310-829-2044 . Yet they are not.page 43 of 86 when it asserts itself as an independent function.Adventures In Quality Assurance . wait a minute! you are probably saying at this point. No one pauses in puzzlement at the fact that the Quality Assurance function is named “Quality Assurance”. The QA function has become. its own needs. saying what Engineering cannot say directly. and so imposes on QA definitions of product quality that are purely notional and impossible to operationalize. then. of course not. then that definition could be used to judge objectively the work of Engineering. Engineering has no interest in either. Nor could it even if it were ideally constituted and practiced. It is important to understand that this institutional domination of QA by Engineering requires that no real definition of product quality exist. in effect. If the purpose of QA is not “quality assurance”. Who does? Engineering.
By Niall Lynch – verlandosta@yahoo.Adventures In Quality Assurance . opens the door to understanding the true nature and role of software quality assurance. furthermore.com 310-829-2044 .page 44 of 86 question that.
the corruption is spread around fairly evenly.Adventures In Quality Assurance . and everyone likes to bemoan these facts.” The cynical option accepts that professionalizing QA is a pipedream that offers only theoretical benefits. It is difficult to dispute that the cynical option has a lot to recommend it. You just need to know the right people. and that’s still how they’re made. and say.com 310-829-2044 . The Standard System is like a country that has lived under a corrupt and inefficient government for centuries. today. and prefers to grab at the very tangible benefits the Standard System dispenses right now. Nevertheless. and things take care of themselves. All in all. so that everyone gets at least a little taste. and assuage any lingering sense of doubt or guilt by posting the odd Dilbert cartoon on the wall of their cubicle to show they have not sold out to the system. The inefficiencies of the system mean no one has to work all that hard to stay in the game. so really there’s no point in trying to change it. We may look at the Standard System. in all its fiendish glory. we must either have recourse to cynicism or to hope. but that’s still where millions are made. “Yes. What’s not to like? The problem with the cynical option is that it sabotages the future of software development. not a bad life. By Niall Lynch – firstname.lastname@example.org 45 of 86 The Lost World If everything I have said about the current deplorable state of Quality Assurance is true. It’s not as though you have to actually be good at something in order to succeed. Everyone knows it is corrupt and inefficient. eminently pragmatic and still leaves one with the ability to complain endlessly about the system one secretly accepts. grease the right palms. It is empirically sound. The cynical person will content herself with this bargain. even if they have.
It just has to be cheaper. incomplete and riddled with bugs anyway. First World engineers may scoff at this possibility. It’s all well and good to muddle through. outsourced software doesn’t have to be better. feudal systems can last a long time as long as they don’t have to compete on cost. First World software engineers have delivered abysmally low levels of quality.com 310-829-2044 . Consequently. for decades. whatever benefits they may deliver. but it is a very real one.Adventures In Quality Assurance . Why? Because. Corrupt. they are actually quite expensive to maintain. you not only begin to lose. Where once outsourcing to Third-World countries was only whispered about.page 46 of 86 This kind of feudal society can continue for a long time. now it is being widely discussed as a serious option. This has already begun to happen in the software industry. Because. but each loss also erodes your ability to ever win in the future. why not pay one quarter the price for it? This is the simple fact that all of the huffing and puffing among First World software engineers in re outsourcing conveniently ignores. as it has in the world of software development. in technical or quality terms. as long as you only have to deal with yourselves. The minute you must compete against more efficient societies. If it’s going to be late. What many First World engineers don’t seem to understand is that the outsourced software doesn’t have to be better. so it’s not as though the bar is set all that high to begin with. The first is competition from more efficient societies. But sooner or later two things are going to happen that make the feudal way of doing things progressively fatal to the society in question. or tried out almost as a lark. This is the main problem with the First World By Niall Lynch – verlandosta@yahoo.
Another way to state the problem is that the social solidarity within the Standard System between upper management and their product delivery organizations is breaking down. are now tempting upper management with even greater rewards than they can eek from their highly-paid. Even with desktop software. if they can get at least the same level of inefficiency for less money. why would they not jump at the chance? The First World software industry is like an hermetically sealed economy that has just woken up to NAFTA. Consequently.Adventures In Quality Assurance . The second factor that is making the Standard System untenable is the exponential growth in both the complexity of the problems software is being called upon to solve.com 310-829-2044 . it does so at a very high cost. Therefore. they no longer see any benefit in being complicit in those failings. in most cases it can be safely ignored. low-performing software development groups. Many First World software development organizations are like the dinosaurs ten seconds before the asteroid hit: Supreme in their world.page 47 of 86 software industry. at least from the point of view of being able to By Niall Lynch – verlandosta@yahoo. since that is what most commercial software companies historically started out making. and the complexity of the software required to solve them. desktop software. Recall that most software development strategies and techniques were developed for the creation of single-user. Suddenly it has to compete in terms of factors it could comfortably ignore before. and supremely oblivious to their imminent extinction. and the rise of India and China as technical powers. there is quite a lot of complexity involved. Globalization. cost efficiency becomes its main Achilles heel. On the contrary. However. one that is increasingly being noticed. Though it generates plenty of money for those involved.
most software development organizations have no clue how to handle them. The problem arises when the level of complexity left latent in the software development process begins to exceed the manifest workings of the development process itself. Rather.com 310-829-2044 . Many companies have in fact been destroyed by their efforts to transition from desktop to enterprise software. Because of their nature.page 48 of 86 make money.Adventures In Quality Assurance . By “complexity” I don’t just mean that client-server software contains more features. Performance is an emergent property. Consequently. we begin to see truly catastrophic failures of the development process. and in the products it produces. I mean something much more significant. Nor can they be tested in isolation from the rest of the system. Enterprise software confronts any software development organization with the problem of emergent properties. This is one way of describing what happens to most software companies when they decide to augment their product portfolio with client-server products aimed at the enterprise. This occurs because the level of complexity in a client-server system is exponentially higher than in a desktop system. When this occurs. since the profit margins are so very much higher with enterprise software. which is usually the case. as is reliability. Yet a desktop software company’s first enterprise project is always a study in tragedy. This evolution is in itself inevitable. These are properties of a software system that only manifest (or “emerge”) when the entire system is integrated and working together over time. emergent properties cannot be coded as separate features. This is so because of what I like to call the “feature fallacy” of software development. not just at the individual desktop user. Software tends to be looked at by all involved in By Niall Lynch – verlandosta@yahoo.
I have seen software companies spend twice the calendar time and person hours trying to fix performance and reliability issues in the field than it took to code and test the initial release in the first place. the mindset of most software organizations is that you first verify all the features. and this is how it is tested by QA. testing for them is prioritized last. This is how software is specified by Product Management. Bugs in emergent properties are not only the most difficult to find. they are the greatest threat to schedule compliance. this is how it is coded by Engineering.page 49 of 86 creating it as a collection of isolated features. for reasons I have already discussed at length. Which is odd if you consider they are the greatest risk. By Niall Lynch – verlandosta@yahoo. and once that’s done. due to the complexity of the testing required to uncover them. would you want to spend a day or two testing performance and reliability.Adventures In Quality Assurance . Naturally. This is why. Or. nor developed. Software is viewed and developed in a very atomized fashion. Most software development groups deal with the problem of emergent properties by ignoring them. and only if you still have time. if they think of them at all. It should come as no surprise. but also are the most difficult to diagnose. and the most difficult to retest. historically. therefore. It is not viewed. and tend to fail catastrophically on the first release of a company’s first enterprise product. this experience does not actually change anything about how emergent software properties are approached in development.com 310-829-2044 . holistically. Consequently. that performance and reliability are the two weakest links in enterprise software. the most serious software problems manifest during system integration testing. Nevertheless.
time and money. Companies that succeed at this boundary are the ones that tend to continue evolving in the marketplace. They don’t really solve the problem at a conceptual or process level. data fusion. and therefore enhances the powerbase of those running software development on such projects.page 50 of 86 This first complexity phase boundary is normally the crucible that winnows out the weakest software companies and sends them crashing back to the desktop world. This is occurring because industry at large has already invested heavily in acquiring enterprise-wide software systems. data normalization. one that involves the same kind of exponential increase in complexity as that which lies at the desktop/enterprise boundary. since the intractable nature of the problem is precisely what justifies demands for more staff and money to address it. operational By Niall Lynch – verlandosta@yahoo. is simple. and now finds itself facing the problem of how to get these disparate enterprise-wide systems to operate together. Rather. and to generate an overall picture of the enterprise’s operations.com 310-829-2044 . and normalize the operation and output of any number of other products at once.Adventures In Quality Assurance . in spite of chronic problems in the area of emergent properties. The software industry is now increasingly being asked to create software systems that integrate. as usual. the industry is now faced with a second complexity phase boundary. This second phase lies at the boundary between products and meta-products. And they do this for each and every release. they power through the problem. Recall from my previous analysis that this is not necessarily a bad thing for those involved. using massive amounts of manpower. manage. How can this be so? How can such companies succeed if they are still struggling with this fundamental problem? The answer. Therefore issues of interoperation. However.
com 310-829-2044 . it is at this second complexity phase boundary that the industry finally reaches a level of complexity where it simply cannot “power through” with hordes of bodies and years of development time. and which may indeed not actually exist at the time. The second is a question of how to develop software so that it can interoperate with and manage the output of software systems that have not yet been developed. Which means in turn that the software industry is being asked to solve two very challenging problems: that of meta-emergent properties. The first is a question of how to manage the emergent properties of a meta-system.page 51 of 86 transparency and decision support are first and foremost in the minds of corporate and government customers. It is important to understand that such large clients are not simply asking for this kind of meta-productizing of the systems they already have in place. They are asking for a meta-product solution that will work with any new enterprise systems they have not yet purchased. and how those metaemergent properties might inflect the whole range of emergent properties throughout its client enterprise systems. They threaten its continued prosperity. The industry has finally reached a point where it must stop and think. It must actually solve these problems at a fundamental level. It can no longer muddle through on a wing. Moreover.Adventures In Quality Assurance . They also threaten its By Niall Lynch – verlandosta@yahoo. at least for those involved directly in product delivery. a prayer and a web-distributed patch. These two problems together pose a huge challenge to the survival of the software industry in its current form. and that of speculative software development. something it generally hates to do. Because the amount of bodies and time necessary to solve these problems approaches infinity.
For these reasons. and what it isn’t. By Niall Lynch – verlandosta@yahoo.Adventures In Quality Assurance . if the industry cannot deliver the kinds of solutions the marketplace is now demanding. and from bonus to bonus.com 310-829-2044 . and actually spend some time thinking about how it does what it does. In particular. it is entirely in the interests of the software industry to stop rushing headlong from release to release. it needs to get a grip on what software quality really is.page 52 of 86 continued relevance.
and so no meaningful comparisons can be made of QA performance across the board. Consequently. Moreover.page 53 of 86 The Cloud of Unknowing If. Having created QA as a separate department. but is merely a project function. In many software organizations. This only perpetuates the feudal relations between the two groups. since the QA staff on each project has to reinvent the wheel each time. It works against not only the interests of QA. By Niall Lynch – verlandosta@yahoo. The first one would be to institutionalize QA as a separate department within your organization. this creates a situation where no meaningful baseline for QA performance can be derived. where do we begin? Certain remediations will be clear from the preceding analysis. with all QA staff reporting up to the head of each project. There is really nothing good to be said about this type of QA organization.Adventures In Quality Assurance . There is no real improvement in QA’s professionalization if the QA department still reports to Engineering.com 310-829-2044 . but also against the interests of project stakeholders at all levels. This creates in turn a situation where the work of QA on each project is not standardized. you must then also make it hierarchically independent of and equal to Engineering. QA does not form its own department. since each group is working independently. then. This system is highly inefficient. we accept the option of hope rather than of cynicism and commit ourselves to remedying the distortions inherent in the Standard System with respect to Quality Assurance. with this type of organization it is impossible to enforce standardized test procedures and methods.
please. To define QA’s role as that of simply “running tests” is not the same thing as defining the unique role of QA in the product development process. just about anyone can run tests.com 310-829-2044 . Rather. Any reasonably By Niall Lynch – verlandosta@yahoo. By the term role I do not mean a set of activities that QA happens to be tasked with performing. One way to define a function’s unique role is to define what it is that those in that role are uniquely responsible for. because people accept that Engineering and Marketing are disciplines. then. After all. We have nothing more to say to one another. Product Management is a good example of this situation. It is as simple as that. One never sees the same phenomenon with respect to writing code or creating marketing plans. and is uniquely responsible for. put this book down right now and go about your business. as long as responsibility for the outcome of those activities cannot be delegated outside the role. One can have a unique role. It is not uncommon. The QA department needs to report to whomever Engineering reports to. Then there is the issue of responsibility and accountability.Adventures In Quality Assurance . Another way to state the issue is to say that QA must have a clearly defined role. when a project is in crisis. All the process in the world will not give you properly functioning QA if QA is not itself first properly institutionalized and organized.page 54 of 86 where QA can only be and do what Engineering wants it to. even if the activities that fall under that role could theoretically be performed by others. I have in mind by the use of this term something that QA uniquely knows how to do. not simply activities. If you are not willing to do these first two things. for people from other functional groups to be drafted into running tests.
In the case of Product Management. haven’t you answered the question about QA and testing? Sure. well. In the example of Product Management.page 55 of 86 intelligent software person can. We can distinguish the writing of product requirements. from taking responsibility for the adequacy of product requirements because we can point to some larger goal that both must serve.Adventures In Quality Assurance .com 310-829-2044 . learn how to write well-formed. Writing requirements is but a means to achieving that goal. we can say that QA’s unique role is running tests. that larger goal is creating a product that will sell into a particular market. only a Product Manager can be accountable for the adequacy of those requirements to the needs of that product’s target market. This reasoning is outwardly logical. but it misses a central point. of course!” is the obvious answer. at the right time. it’s all the tests passing. Therefore. and someone who is accountable for creating the right requirements. but ultimately it’s QA’s responsibility for the outcomes of those tests. And what is product quality? “Er. But what exactly is the larger goal of executing software tests? “Well. a goal that exists apart from the activity under discussion. quality. All right then. someone is probably saying right now. with a small amount of training. However. for the right market. logical. internally-consistent and actionable product requirements. the argument is not circular because the goal of PM is not defined as “completing By Niall Lynch – verlandosta@yahoo. not an end in itself.” Here we realize the essentially circular nature of the definition of product quality. This is the key difference between someone who is just good at writing requirements. anyone can run tests. Accountability always lies with those responsible meeting of that larger goal. for example.
The criterion for effective product requirements is not they were all implemented. that such questions are actually asked. Another reason for this difference is that in the case of product requirements we can ask reasonable. a definite. they conquered or created a new market for the product in question. 10 We can in fact test their adequacy rather directly before they are ever accepted or implemented simply by talking to major customers or by looking at market surveys and other market data generated by outside organizations. measurable outcome in the market place. there is no corresponding notion of adequacy. well-formed questions about their adequacy. How could there be. when there is no larger goal to the activity of running tests? One of the greatest.Adventures In Quality Assurance . but as those requirements having. of course. By Niall Lynch – verlandosta@yahoo. in addition. and most persistent. but rather that. yet which nevertheless demonstrate very poor quality in the field. There is no larger goal that can be used to determine whether the passing of a certain set of tests accomplished what QA is uniquely responsible for. Yet in the case of product quality. with every test having been passed.com 310-829-2044 . This is one of the key reasons that hundreds of software products are released every year.page 56 of 86 requirements”. We define product quality as a set of tasks (running tests) having been successfully completed. unknowns in almost any software testing project is whether the set of tests being run is adequate to the quality problem posed by the product. Yet this is not what we say about product quality. 10 Which does not mean. having been implemented.
then the accountability for choosing to ship a product with a certain well-defined level of quality also becomes a shared choice.page 57 of 86 Indeed. notion of quality. precisely so that it cannot function as a fulcrum for analysis and accountability. For if quality becomes something transparent. because there does not exist a general concept of adequacy that could be used to establish its comprehensiveness. But this upsets one of the pillars of the Standard System: that of putting all accountability on QA.com 310-829-2044 . No one really wants to know what the By Niall Lynch – verlandosta@yahoo. The lack of a testable notion of adequacy in Quality Assurance is the major artifact of its non-prefessionalization. Yet this is really all “software testing” is in most software development organizations. even as any real authority is denied to it. truly testable. In fact. The most dire consequence of this situation is that software development has no meaningful. Indeed. and therefore has no clear concept of the content of its own expertise. and not the definition of a meaningful goal. many software testing organizations would instantly be struck mute.Adventures In Quality Assurance . QA’s feudal servitude to Engineering has prevented it from developing a clear concept of its own role. the definition of product quality is purposely kept as vague and ideal as possible. hitting buttons and finding bugs is not the definition of an expertise. As I have pointed out before. something knowable by all involved. as I have pointed out. not the definition of a role. In most product delivery organizations. if asked to demonstrate this adequacy. one of the dirty secrets of software testing is that most QA groups test what is easiest for them to test. the adequacy of the “test plan” must be taken largely on faith by the other functions. not what is most necessary for them to test.
but the industry as a whole. Even after that has occurred.Adventures In Quality Assurance . Keeping this in mind. but not a sufficient step in professionalizing the QA function. of a meaningful concept of product quality. I will devote the remainder of this inquiry to filling in that hole for you. This means.page 58 of 86 true level of achieved product quality is. because that plausible deniability is what protects them from the consequences of poor quality. that institutionalizing QA as a separate department. free of and equal to Engineering. dark black whole remains to be filled in. in turn. is a necessary.com 310-829-2044 . the hole that marks the spot where a proper understanding of QA’s unique expertise should be. it becomes sadly clear that we cannot professionalize QA simply by reversing the most obvious injustices of the Standard System. This is not sufficient because one of the main results of the Standard System has been to deprive not only QA as a function. a deep. By Niall Lynch – verlandosta@yahoo. We cannot magically derive it just by reversing the mechanisms that have prevented its appearance.
it is entirely possible to achieve very good process efficiency with consistently low product quality. Customer satisfaction is not really a single thing. the quality of technical support. Moreover. but because they can be defined independently of one another. The difficulty arises when we realize that customer satisfaction rests on a far broader base. by “product quality” I do not mean process efficiency. Let’s begin. I think it would be better if I derived those definitions from the basic realities of commercial software development. By product quality here I do not mean customer satisfaction. licensing terms. Though it needs to be said that one of the key parameters By Niall Lynch – verlandosta@yahoo. then.page 59 of 86 Laughter in the Dark I could just tell you what real Quality Assurance was. but it cannot be defined from so general a perspective. Not because the two cannot be related in mutually signficant ways. especially if those conclusions may at first seem novel or out of the ordinary. Product quality is an important component of that aggregation. very consistent product quality with low process efficiency.Adventures In Quality Assurance . whether the product’s feature set meets their needs. etc. by saying what we don’t mean by “product quality”. including the quality of the sales experience. Indeed. people tend to accept truths more readily if those truths are conclusions they themselves have reached. Moreover.com 310-829-2044 . though the two are related in significant ways. but an aggregation of all aspects of quality throughout the entire business process. You can certainly achieve very high. So rather than just lay out what I think QA actually is and does. that is one of the key features of the Standard System. but that would be something of an anticlimax. On the other hand.
I must also repeat that by “product quality” I do not mean pre-release bug metrics. The obesssion with bug metrics enforces a minimalist. The focus on bug metrics also distorts our understanding of product quality because it leads us. negative definition of product quality that itself needs to be overcome if product quality is to be correctly understood. as the absence of things going wrong. willy nilly. By Niall Lynch – verlandosta@yahoo. bodies or schedule time. and to what degree of depth. for example. to define product quality negatively.page 60 of 86 necessary to calculate process efficiency can only be provided by a correctly functioning QA group: the actual level of product quality achieved. in terms of dollars. bug metrics by themselves do not validate the adequacy of the test coverage. Having zero open bugs 11 tells you nothing about what the testing actually covered or left out. since many bugs are ‘closed’ for release. Bug metrics simply a measure of QA effort. as I have pointed out already. and this is the case for product quality as well. even though they have not been fixed. For without knowing that. it is easier to illustrate common sense in software development by appealing to our experience outside of software development. then pre-release bug metrics are far too restricted to enlighten us. not of quality itself. it is impossible to calculate the cost of achieving that level of quality. For some reason. Yet just because we have not found something obviously going wrong. nor even investigated.Adventures In Quality Assurance . If customer satisfaction and process efficiency are too general to provide a good definition of the term. doesn’t mean quality is present in the product. you have Even this case needs to be defined ideally. Suppose. This is so because. etc.com 310-829-2044 11 .
convinced their fantasy car was a piece of crap.Adventures In Quality Assurance .com 310-829-2044 . a piece of trim falls off in your hand. imagining how you’ll feel driving the car to the next Linux User’s Group meeting. You have won the lottery. Unfortunately. sitting in the supercar of your wildest fantasies. nothing going wrong doesn’t really tell you anything about the quality of the car either. Should you buy the car? Some people at this point would storm off to the Lexus dealer. While you wait. because having a piece of trim fall off tells you nothing about the actual quality of the car. The dilemma is that you simply can’t find out enough during a test drive to determine the true level of quality of a car. Others would pretend it hadn’t really happened and would quietly try to reattach the piece of trim. Unfortunately. You now have a pile of cash. But would they be right? The reality is that in both scenarios any conclusion would lack an adequate basis. These are usually the sales people. waiting for a sales rep to notice you. You are at your local Ferrari dealer. and are able to go out and buy the car you’ve always dreamed about. that nothing obvious had gone wrong during your inspection of the car. so you must base your decision on accidental details that may or may not have any greater significance or predictive power. you fiddle with the controls and feel the materials of the seat and dashboard. Suddenly.page 61 of 86 hit some kind of jackpot. broken the bank in Vegas or sold your company’s stock just in time. These people are usually the engineers. before the dealer noticed and charged them for it. the same is By Niall Lynch – verlandosta@yahoo. Would that mean you should buy it? Many people would think so. complaining about shoddy workmanship. Suppose again that the piece of trim hadn’t fallen off. Congratulations! You’ve found a bug.
but you could admit it was a well-made car. at least enough to keep you from dozing off. even though none of them wears a cat suit. and since you are 35 years old. even if it is not a product that you are necessarily interested in. while you want to see an action adventure movie full of exploding cars and imploding heads. you are nevertheless able to admire it in an abstract way. This prosaic example illustrates something very important about product quality: It can be perceived independently of your personal likes or dislikes. It is possible to determine that a product has quality. Your significant other wants to see an historical drama. even though it wasn’t the kind of movie you would have seen on your own. You watch the movie with your significant other and. but you can still tell that Celine Dion has a good voice. You may find driving a Lexus far too dull. for example. Since your signficant other is the first one you have ever had. or that meets your specific needs. You may not like pop music. you can honestly tell your date that it wasn’t bad. all by itself. The characters are diverting. The easiest way to get at a good definition of product quality is to ask ourselves how we define product quality in other areas of life. The plot has some interesting twists and turns. The production values are very high. The cinematography is stunning.Adventures In Quality Assurance . When it’s over. even though you find this type of movie tedious and silly. Suppose you are taking a date to the movies. By Niall Lynch – email@example.com 62 of 86 true of most software testing. Let’s take movies. is simply not an adequate basis for determining product quality.com 310-829-2044 . you come to the reasonable conclusion that it’s best to see the historical drama. Finding or not finding bugs. even though it doesn’t really tickle your fancy. and has subtitles to boot. The acting is good.
at least in the examples cited. They define concrete achievements that must be manifested in the work or object in question to be defined as “good”. “I can see how X satisifes the requirements for X. This practice continues unchallenged precisely because. we can determine what objective requirements have to be met in order for those products to be considered ‘good’. that exists independently of our desires.com 310-829-2044 .” These standards are also positive in nature.”? Never. but it’s not something I like. very ironic to give such a review. and the reels were shown in their proper order. If this is true. the air conditioning was working fine in the theater. We can distinguish between product quality and our own satisfaction with a product because. Yet this is basically how software quality is defined in most organizations. unlike in the case of films. there really is no positive standard of quality. the soundtrack was in sync with the action.page 63 of 86 You may look awful in a bikini. some specification. then product By Niall Lynch – verlandosta@yahoo. All right then. There is some standard. right? In fact. but those requirements don’t define something I like or need. ‘X was good. When was the last time you heard a movie reviewer say. “This is a really good movie that you must see because the projector never broke down. you would think such a reviewer was either a fool or very. that we can use to judge the adequacy of what we are experiencing.Adventures In Quality Assurance . what is a positive definition of product quality? In the case of a commercial software product. you might be thinking now. but still be able to see how others might look just fine. that objective standard can only be one thing: the product’s formal requirements. When we say.” we are really saying.
There is another important difference. or does not operate in all the environments By Niall Lynch – verlandosta@yahoo. It is truly a sad experience for a Product Manager to discover that a product does not in fact contain all the features specified. and still not meet the requirements defined for it. What the vast majority of software testers actually test are code functions. It is entirely possible for a product under test to behave the way Engineering thinks it should.com 310-829-2044 . modules and subsystems. logical and necessary. This may sound like a fine distinction. This is. functions. actually meets the requirements it was written to fulfill. This is a key point. as written. Most software testing efforts are organized around the internal structure of the code itself. Defining product quality as conformance to requirements shifts the focus away from code. It is far more stringent than simply determining whether something goes wrong.Adventures In Quality Assurance . it is also necessary to make the separate determination whether the code. in the sense that I have described it. and to the need that the product is supposed to meet in the marketplace. Why? Because in addition to determining whether the code behaves as Engineering thinks it should. But it is in no way sufficient to assure product quality. but in reality it is not. Note how different this standard is from that of finding or not finding bugs. parameters and outputs.page 64 of 86 quality in turn can only be one thing: demonstrable conformity of the software under test to its formal requirements. Product quality. the structure of the test effort mirrors exactly the structure of the code itself. In other words. is a positive ascertainment of capabilities. I have personally seen this happen on many occasions. of course.
that the product Though to be fair in unfairness. not validation. other functions have no choice but to default to the technical specification created by Engineering. With those terms and definitions and mind. Let us call the process by which the QA group determines whether the software behaves according to the Engineering spec verification.Adventures In Quality Assurance . By Niall Lynch – verlandosta@yahoo. Let us call the process by which the QA group further determines whether the software under test. In the absence of well-formed. it also needs to be pointed out that often even a technical specification is lacking. This is another reason why bug metrics alone tell you nothing. and to discover this only after the product has been released. at best. then the product development process has no way of determining whether the product being produced will meet the needs of its intended target market. This is one area where Engineering is itself victimized by the “need for speed” inherent in the Standard System. To be fair.page 65 of 86 specified. they are telling you whether the software behaves as Engineering thinks it should. and its technical specification. fulfill the product requirements validation. in turn. a lack which turns QA into a form of fortune telling. If no validation is being performed. Yet this is not an unusal occurrence. much software testing is code-oriented because there are no formal requirements. Which means.com 310-829-2044 12 . just so that the product can be made. It needs to be acknowledged that Engineering is often called upon – unfairly – to supply what Product Management has failed to. 12 At this point it is useful to introduce some more specific terminology to help us express the point at hand. we can say that most software testing efforts only perform verification. because. comprehensive requirements from Product Management.
” Not a whole lot of bang for the buck. and sadly real. Yet this is almost never the case. This is obvious if we accept that it is QA’s job to tell Product Management whether its requirements are being met. resemblance to My Fair Lady. “Well. we’re pretty sure the product does what we think it’s supposed to do. to be jealously guarded against any influence from outside Engineering.Adventures In Quality Assurance . is it? There is a final general conclusion to be drawn from the definition of product quality as demonstrated conformance to requirements. The captive nature of QA. the inability to distinguish between verification and validation: All these circumstances conspire to create a situation where the best one can hope for from software testing is some form of adequate “code coverage”. is what has kept QA from developing such a strong interface with Product Management. of its tendency to treat QA as its property. since any attempt to do so is commonly seen as a weird kind of treason or unfaithfulness to Engineering. and it By Niall Lynch – verlandosta@yahoo. and this institutional reality. though few of those involved are fair. the Engineering-QA relationship has a sad. the notion of “code coverage” is a quality criterion often invoked by engineers. In this respect. the lack of well-formed requirements. All most product development efforts can do is say.page 66 of 86 development process is eating up huge amounts of time and money. This attitude. or ladies. QA must have an interface with Product Management that is every bit as robust and authoritative as its interface with Engineering. without being able to do the one thing management really wants it to do.com 310-829-2044 . Recall my earlier description of Engineering’s possessive attitude toward QA. Indeed.
that bedevilled the bug metrics criterion. One hopes an engineer has done this before submitting any code for test.page 67 of 86 is not surprising.com 310-829-2044 . 13 13 This is not to say that running each line or module of code is not a useful basic test at the engineering stage. Chief among them is: What do you mean by “coverage”? How do we know when code has been adequately “covered”? Has code been “covered” when it has been run once? Twice? Three times? In which environments? In what temporal order in relation to other code modules? According to which functional trajectories? Etc. However. and obviously. constitute quality assurance.Adventures In Quality Assurance . By Niall Lynch – verlandosta@yahoo. Code coverage belongs to verification. the code coverage criterion really begs more questions than it answers. and the same failing. You will notice that this is the same question. Neither criterion provides a verifiable notion of adequacy. the whole notion of code coverage is every bit as ambiguous as the criterion of bug metrics. There are no ready answers to any of these questions. “code coverage” is also an inadequate quality standard. But you still don’t know anything about the quality of the system as such. that this would be their default standard. it’s a useful thing to do. in light of the above. Relying on the code coverage criterion is a little like thinking you have tested the quality of the plumbing system in your new house by flushing all the toilets. First. Though it sounds like a simple idea. “code coverage” tells us nothing about requirements coverage. and cannot take its place. But this procedure does not. by itself. for two reasons. and therefore fails as a criterion for quality. not validation. and it may ease your mind about each individual toilet. Certainly. Second.
and Cape Canaveral. paradoxically. Many books on the subject are written as though software products are developed by a clandestine Communist society located in Silicon Valley. Consequently. however sensible they may be on any specific topic.page 68 of 86 The Parent Trap So far we have determined two things about the nature of effective quality assurance. This is in itself a fairly radical step. we must import into our definition of quality assurance the distinction between verification and validation. to turn our attention away from the subject of quality assurance itself. Yet. the immovable objects that secretly define QA’s By Niall Lynch – verlandosta@yahoo. with outposts in Washington D. in some systematic fashion.Adventures In Quality Assurance . and knowing that the code written by the engineers meets the requirements defined by Product Management. First. steps that bring us to the threshold of an even more fundamental truth about product quality.C. the product requirements. a society from which the imperatives of quarterly financial results have been banished. in order to do this. second. These are two large and important steps in our inquiry. that “product quality” consists of verifying. many software process books have an unreal quality about them. like girls from a treehouse. between knowing the code behaves the way the engineers think it should. and examine in more detail the corporate financial imperatives it must co-exist with and adapt to on a daily basis.com 310-829-2044 . as any rank and file quality assurance person can tell you. The easiest way to uncover this truth is. and that. since one can read books on software quality assurance by the score without ever stumbling across a serious analysis of this topic. these financial realities are the reality of quality assurance.
these financial imperatives must be placed at the heart of any real understanding of how to assure product quality.com 310-829-2044 . it is the situation faced by almost every product delivery team. Yet this is certainly not true in either case. Yet accepting this also turns out to be the key that unlocks the door to a real understanding of product quality. Indeed. Nothing about the way publicly held companies raise and maintain their stock prices is going to change any time soon. Yet the quality of the product is not good enough to ship. unhelpfully. This is not surprising when we realize that many software processes were developed either by the US Government or by very large companies developing software for internal use. Because this is true. we must accept in turn that. or. the financial demands of a publicly held company are treated as extraneous to problems of process. or its stock will suffer immediate and perhaps irreparable harm.Adventures In Quality Assurance . no miraculous wave of compassion and fiscal forebearance on the part of impatient investors.page 69 of 86 native habitat and its survival strategies within it. which is also incidently the committed release date of the product. What do you do? This is the situation that every QA manager and project lead faces throughout their careers. at least for publicly held companies. By Niall Lynch – verlandosta@yahoo. demands that will disappear if process is done correctly. There is going to be no magical cancelling of the quarterly financial reporting calendar. time and time again. Yet. at best. Because of this quirk of history. this is a situation that is usually abstracted out of many software process improvement recipes. The company must make this ship date. For is it not true that the perennial software quality assurance dilemma is as follows: The end of the company’s fiscal quarter is approaching. not for external customers.
Just do it isn’t just the motto of a sport shoe company. One hears this complaint time and time again from both engineers and QA staff. This creates a situation where everyone in product delivery is sent rushing pell mell.page 70 of 86 How can this be? The existence of a “hard date”. demanding that the date be met. in whatever shape it happens to be in. These are then almost immediately thrown back at product delivery management with the curt demand. is always presented as the one thing software development people wish they could do away with. and the project slides into a black hole of unknowability. “You must make this particular date. to “make the date”.com 310-829-2044 .Adventures In Quality Assurance .” Product delivery management then dutifully prunes the feature set in order to meet the “hard date”. It is in the midst of this frenzy of activity for its own sake that any rational project tracking practices tend to get cast aside. Oh. Product delivery management is often caught in a cruel game where. though this is itself usually By Niall Lynch – verlandosta@yahoo. And for another thing. only to emerge at the very end. Everyone “just does it”. for sure. this complaint is difficult to fault. with “best in class quality”. period. like Keystone Cops. at the beginning of a project they are asked by upper management to provide detailed project schedules and firm product release dates. furthermore. this time with Sales baying at the moon in the background. one which was selected in the first place for purely financial reasons. one that cannot be changed without catastrophic consequences and. with no new staff. and by the way. On the face of it. Upper management again throws it back. with the originally stipulated feature set (otherwise there is no reason to have a new release). It “goes dark” from an in-process perspective. and resubmits their schedule.
and one that seems to have little to do with the necessities and realities of the product development process itself? Are they just being dictatorial and clueless.Adventures In Quality Assurance . However. It should be obvious based on my previous analysis of the Standard System that a project’s slipping as quickly as possible into an unknown state benefits everyone involved. the date is met and the company is saved. Yet even from the point of view of the product delivery team. to be determined later by customers. both politically and operationally. who really wants to deliver it? “No one” is the answer to both questions. What good does it do for you to be able to know that the project is weeks late? Or that quality coverage is going to be spotty at best? Or that major features are only going to be half-coded by the ship date? Who really wants to hear that news? More importantly. There is no upside to transparency. to purely political coping strategies that only exacerbate the problem. but they are nevertheless very serious. it is clearly dysfunctional to pretend that the hard date problem is a nuisance to be done away with. Ask yourself: Why does upper management constantly insist on a hard date. if all it will reveal is that things are going wrong to an upper management group that doesn’t want to hear about it. for the simple reason that the ways in which product delivery responds to that problem are inherently unsuited to coping with it successfully. The operational problems are obvious. and continues to lead. The political ramifications of a purely political accommodation with the hard date problem are not quite so obvious.com 310-829-2044 . Ignoring the problem on an operational level has led.page 71 of 86 the Big Unknown. which is the standard diagnosis from the product delivery side? By Niall Lynch – verlandosta@yahoo.
what do they have to lose by providing these through pure stipulation? If you have children. perhaps not. and will have it done by the end of the day.” OK. Your child says.. Your child responds they will get to that just as soon as they are done with their homework. “We’ll have everything done at the end. You ask your child why it’s not mowed. At this point. Then again. but for now you’ll give them the benefit of the doubt.e. and then we’ll be done. A more accurate analysis of the hard date phenomenon leads us to the simple logic of expectations. Often what you hear is. the better to remove another excuse for not meeting it.). i. and I didn’t want to mow the lawn late at night ‘cuz that would wake up the neighbors. so I had to work later. If your product delivery group consistently fails to make dates. if it consistently fails even to be able to provide key commitment points within the development schedule. my homework was harder than I expected. and the lawn greets you in all its unmowed glory. 14 what possible confidence can any executive have in the preferred date options offered by product delivery? And if they can have no confidence in the product delivery group’s ability to generate its own reliable dates and milestones. Surprising as it may seem. That is at least a rational explanation. You wake up the next morning.com 310-829-2044 14 . engineering staffs often refuse to make hard and fast commitments to internal milestones. “Well. You sense your child might be playing you. you should be familiar with this logic.Adventures In Quality Assurance .page 72 of 86 Perhaps. uh. you’re willing to take a small amount of the blame on yourself for not taking certain realities into account. Suppose you want your child to mow the lawn. By Niall Lynch – verlandosta@yahoo. You tell them to please mow the lawn by sunset today (you have picked up on the fact that you need to make your deadline more precise.” Not very confidence inspiring. a commitment to delivery a particular subset of functionality at a particular date or date range.
Too many obviously phoney excuses have been thrown at you. over time. you might have been sensitive to this very real obstacle. and at this point you really don’t care. No questions. I don’t care what the consequences are to your schoolwork. and so no one wants to hear about it now. In fact. no excuses. my homework was pretty heavy today too. uh. So I’ll do it tomorrow morning. This is the basic process in software companies. and I knew that would clog up the mower. you realize having to suffer a little as a result of laziness might make your child more willing to do your bidding the first time. These arbitrary deadlines may be obviously problematic in themselves. “Look. but you’ve lost all credibility.Adventures In Quality Assurance .” You glower at your child. But it had just been watered by the sprinkler system. Consistent failure. and you see the lawn is not mowed. In this respect. just after sunset. now convinced you are being played. This time you confront your child a little more strongly.page 73 of 86 You come home from work the next day. you’ve had it now. but I want that lawn mowed by 3 pm tomorrow. but. Why isn’t the lawn mowed? you ask. Why wasn’t the lawn mowed before the sprinklers came on? Why the strategic delay until it was no longer feasible? Ok. the relation between upper By Niall Lynch – verlandosta@yahoo. or else. But you’ve been played too much to sympathize now. I don’t care what it takes. You sit your child down and say. to set expectations correctly and fulfill them leads to the imposition of arbitrary deadlines by the powers that be.com 310-829-2044 . “Well.” Your child becomes very alarmed. I can’t do it tomorrow! I have soccer practice tomorrow afternoon! If I miss that I might get cut from the team!” Earlier in the history of your lawn mowing crusade. but I still tried to mow the lawn like you asked. “But.
Yet. Quality can only be this platonic ideal. Yet. regardless of the political value of adhering to the Standard System’s view of quality. how could it be otherwise? The amount of quality you’re going to get is going to vary according to all the same factors that affect every other major project parameter: expertise. as I have pointed out previously. staffing level. In this respect it is like virginity or coolness: You either have it or you don’t. there is no in-between. instead of trying to define around it or thinking that we could do our jobs right “if only” the hard date problem didn’t exist. This may seem like a startling statement. If we do that. how much money for equipment they have. Perhaps the answer is to grow up? Let’s take a novel approach to the hard date problem. Let’s go further and ask ourselves what our work would look like if we defined it completely in relation to the reality of the hard date. The By Niall Lynch – verlandosta@yahoo. depending on how many programmers you have. etc. we will find that things become interesting fairly quickly. but really. recognizing that a product’s feature set is a product variable. let’s ask ourselves what things look like if we accept it as an unalterable aspect of software development.page 74 of 86 management and their product delivery organizations is exactly like that between an impatient parent and a recalcitrant child. the hard truth remains that quality is a major project variable. regardless of how every other project parameter may vary. for example. the Standard System resists admitting that quality is also and equally a variable. Instead of bemoaning its existence. that never varies. What emerges from this thought experiment in the case of product quality is a rather shocking discovery: Quality is a major project variable. time and budget.Adventures In Quality Assurance . We have no problem.com 310-829-2044 . how much time you give them.
when the final bugs need to be categorized and adjudicated. They prefer to let this reality remain tacit. the consistent insistence by upper management on the most meaningless of quality criteria also has a logical basis. resources and feature set) so beloved of project managers is actually a rectangle. even if that logic is often obscured by the political value of this meaningless. below which you have catastrophe and above which you have needless delays with no financial benefit to the company. As with dates. In practice.Adventures In Quality Assurance . A project team must be able to decide what level of quality is acceptable for any given project. and. as with the hard date problem in general. if ever. and formally latent until the very. Why? Because it is heresy in many organizations to openly discuss the level of quality the whole project team is going to commit to. Nevertheless. product delivery groups are rarely. making gains in product quality becomes cost prohibitive. Quality is a black hole of useful By Niall Lynch – verlandosta@yahoo. very last stages of the project. therefore. and must accept that that level comes at a cost. every seasoned product delivery person understands that. Yet no one really wants to factor this practical wisdom into their projects. since anything less than “best in class quality” is anathema to upper management. This is not so radical as it might seem at first.com 310-829-2044 .page 75 of 86 traditional “triangle of trade-offs” (time. which must also be weighed in the balance. and won’t really have a noticeable effect on customer satisfaction or sales. beyond a certain point. capable of providing meaningful measures of product quality. from the very beginning. in an explicit and actionable way. are unable to provide meaningful decision support data on product quality for upper management to factor into their judgments. Everyone knows that there is a “sweet spot” for product quality.
It must be able to present alternative scenarios of quality coverage. which in turn circles back and further damages operational efficiency. so that each outcome can be meaningfully evaluated in relation to the business goals of the project. and one of the key skills. again. What may seem like willful childishness on the part of upper management is actually a logical result of product delivery’s inability to make meaningful commitments. etc. then one of QA’s key responsibilities is the quantification of quality at the beginning of a project. and consequently quality by stipulation becomes. one of the key responsibilities. resource levels.page 76 of 86 information. product delivery plays into the hands of upper management who. provide meaningful data. and generate meaningful measurements of key project parameters. By Niall Lynch – verlandosta@yahoo. However. We see that a properly functioning QA group must be capable of describing possible quality outcomes in an analytical fashion. The operational failings of each part of the company compound the political opportunism of every other. of QA is the modelling potential quality outcomes. of which quality is one. The breaking begins with the realization that if quality is a variable whose value can be meaningfully defined. In other words. for political reasons. don’t necessarily want to see this data in the first place. This implies in turn the abilitiy to express product quality in terms that can themselves be quantified in the first place. describe the pros and cons of each in relation to other major project parameters (schedule compliance. projected in-field quality.).Adventures In Quality Assurance . the circle can be broken. the only logical choice left to upper management.com 310-829-2044 . By failing to do so.
Along with modelling final quality outcomes as a basis for the project’s overall quality 15 I show how this is done in Part II. Thus. If quality is validation of requirements.page 77 of 86 This means in turn that QA must be able to conceptualize quality coverage. Quite different from its reduction either to purely technological knowledge or to pushing buttons and finding bugs. This is a very different understanding of what QA is and does. But this does not exhaust QA’s unique responsiblities and skill set. then we are able to give more precise definition to what QA’s true area of expertise is. at least in part. the way QA models quality is by defining the quality space for any given requirement.com 310-829-2044 . quality modelling can be defined as fitting various levels of actual quality coverage over the background of the total theoretical coverage that could be obtained given world enough and time. If we understand this. By Niall Lynch – verlandosta@yahoo.Adventures In Quality Assurance . It is the ability to model quality accurately and in a way that provides decision support for the project’s quality commitment. Here is where our definition of product quality as conformance to requirements provides the information QA needs to effectively model quality outcomes. 15 and proposing how much of the theoretical quality space needs to be covered in order to assure the necessary level of quality. it must have a way of breaking down quality coverage into coherent units and sub-units. In other words. then quality coverage can be defined as the depth and breadth to which requirements are validated. and providing forecasts of the likely result on other key project parameters for any given quality optimization. such that the meaning of “coverage” can be precisely known for any proposed quality outcome.
This is a very different emphasis from what one normally encounters. By Niall Lynch – firstname.lastname@example.org 78 of 86 commitments. Software testing is task-based. and at phases of a project when such data may seem worthless. Recall that in the Standard System. one that can only be finalized if an earlier release date is promised. It is the very arbitraryness of the ship date. and the data to back that up. It is the toy surprise rattling around inside the empty box of Cracker Jacks. as it is currently practiced. it may at first seem puzzling what benefit is gained by tracking quality so minutely. Quality This is not an uncommon occurrence. If QA can do that. and the ease with which it can magically shift backward in time. That is to say. product quality is only determined and therefore knowable at the end of a project. then it can provide rational analysis of the risk of shipping on any given day. it must be able to know with certainty whether the quality outcome so modelled and agreed upon is in fact being achieved on a weekly basis. Here we get to the heart of the difference between “software testing”. analysis that can be backed up with real data and well-supported projections of customer satisfaction in the field. Software testing is meant to assure a particular outcome at the end of a project (though it usually fails to do this). especially if a large new customer deal suddenly pops up.com 310-829-2044 16 . Because of this historical orientation. and quality assurance. 16 that makes it necessary for the Quality Assurance function to be able to provide a “ship/no ship” recommendation on any given day. of course. The answer is.Adventures In Quality Assurance . QA must also be able to monitor and report out effectively on inprocess quality thoughout the development effort. Quality assurance is goal based. the very “hard date” problem that we have been considering. not just on the final day of the project.
e. Ok. Test pass/failure data can be translated back into validation of specific requirements. analyze and report out on the validation of the requirements defined for the product under test. and thus cannot provide a basis for effective decision support. QA’s tests themselves must be directly traceable back to specific product requirements. but that data cannot be inherently related to a meaningful analysis of overall risk. how many tests have actually been run). If we accept this to be the case then.. but how? Spin Cycle The first part of the answer to this question consists of referring back to what was said earlier about the definition of product quality: demonstrated conformance to product requirements. This insight leads in turn to the equally logical conclusion that.com 310-829-2044 . can provide an overall picture of both quality state and trends.page 79 of 86 Assurance provides a continuous window into in-process quality throughout the lifespan of a project. Let’s pause for a moment to reflect on how different this is from normal software testing in the Standard System.Adventures In Quality Assurance . therefore. By Niall Lynch – verlandosta@yahoo. from a day-to-day perspective. then we can see how the outcomes of QA’s testing can become accurate information on product quality. If we accept this. Quality assurance can do both. which. logically. when correllated with the test schedule itself (i. the goal of Quality Assurance must be to track. such that the success or failure of any given set of tests means the validation or non-validation of a specific requirement. Software testing can produce data.
in quantifiable terms: 1. tests trace back to code structures and systems. What your operational definition of achievable quality actually is. then the hard date problem is transformed. whereas in the Standard System it is seen merely as the absence of problems or “bugs”. because you have placed that problem at the center of your quality assurance By Niall Lynch – verlandosta@yahoo. Of course tests are still executed in a true quality assurance effort. in my definition the main task of quality assurance is analysis. in my definition QA’s tests must be traceable back to requirements. whereas in the standard system it is defined as an outcome we can only know at the end of the project cycle.page 80 of 86 First. How far you still need to travel to reach that goal.com 310-829-2044 . and foremost. 2. in my definition product quality is a persistent product attribute that exists. Second. but the results of these tests can be related to an overall analytical framework that can actually tell you. and whose level can be measured. not to requirements. whereas in the Standard System it is execution. At any given point in the process how far you’ve come with respect to your operational defition of product quality. Third. Fourth. throughout the project lifecycle. in my definition product quality is seen as something positive in and of itself. the concrete manifestation of specific capabilities. whereas in the Standard System if traceability exists at all (and it rarely does). If you have a system and a methodology in place that gives you all three types of knowledge listed above.Adventures In Quality Assurance . and 3.
Adventures In Quality Assurance - page 81 of 86
efforts from the very beginning. In a software testing culture, you just test as much as you can until the date arrives, and then you hope for the best. When asked whether the product is ready to ship, you may feel that testing has been inadequate, but you can’t really articulate how much more testing would be necessary to satisfy your doubts. You can’t say which specific requirements have been validated and which have not, you can only say how many open bugs you have. Since you have no firm grasp of how testing maps back to requirements, it’s impossible for you to say how much more testing may be necessary, nor can you say how n amount of extra testing will yield x amount of extra quality, and in what product areas. Because of this, you hesitate to press the issue. You satisfy yourself with saying something like, “Well, we’d like to do more testing, but we think it’s OK now,” and cross your fingers. In a properly run quality assurance environment, you would know the answers to all these question, and you would be able to provide real decision support data with respect to product quality. Because you could, you would face two scenarios, either of which you could live with. In the first scenario upper management reviews your quality assessment, accepts the quality risk profile you have provided, and decides to ship the product. If the product in the field then goes on to manifest quality problems in the areas you indicated required more testing, no one will be surprised and you will not be blamed. The executives will have released the product knowing they were likely to have problems in those areas, and Technical Support and Sales have also been briefed and trained to respond to them, prior to the product’s ship.
By Niall Lynch – email@example.com 310-829-2044
Adventures In Quality Assurance - page 82 of 86
This first scenario illustrates one of the major differences between quality assurance and software testing, and their respective outcomes. There is a huge difference between shipping a product, with bright assurances of “best in class quality”, only to have it manifest problems in the field that come as complete surprises, and shipping a product where the same problems manifest in the field, but you knew they would beforehand. Here is where we see how quality assurance is inherently anaytical. It keeps product quality in a known state throughout the process, and can give an accurate assessment of the level of achieved product quality even if that level of quality may not be optimal from other perspectives. Whereas in the software testing model, the controlling idea is that the purpose of the QA group is to achieve one, and only one outcome: “best in class quality”, but is unable to tell you exactly what that is in a non-circular way. The other scenario is that upper management looks at your analysis and quality risk profile, and decides that the risk to customer satisfaction and sales is too great. Because you have provided a detailed quality breakdown by requirement and capability, upper management has the freedom to select specific areas of the product to be fine-tuned for a higher quality level, and is able to know how much more time and money reaching that level will cost them. The result is that the team is given a specific amount of extra time to achieve a specific amount of extra quality. If your team delivers on that promise, then you are in the clear. This second scenario illustrates another important difference between quality assurance and software testing. In quality assurance, quality and effort
By Niall Lynch – firstname.lastname@example.org 310-829-2044
Adventures In Quality Assurance - page 83 of 86
metrics can be correllated and projected with some degree of precision. Whereas in software testing this is not possible, creating a situation where the QA function is unable to say exactly how much time it will take to get to a certain level of quality, or to know that such a level of quality is unattainable within financially responsible parameters. Since QA cannot do this in a software testing environment, upper management has no reason to listen to a “no ship” recommendation. If you are able to provide this kind of risk analysis and decision support, with the data to back it up, you will see a huge change in the attitude of upper management towards ship dates. You will suddenly discover that these blind, irrational, dictatorial authority figures are actually quite willing to make hard decisions about product ship dates, if you can provide them with accurate and timely data on the risk associated with those decisions. You will discover that their relucatance in the past to listen to your inarticulate pleas to delay shipment of the product was due to the unlimted risk you were asking them to take on your behalf, with no reliable assurance that everything would not blow up in their faces. Remember, to executives risk is radiation. You can’t ask them to walk into a reactor without the proper shielding. Of course, the analysis above is somewhat ideal, since it assumes that the engineering function is also capable of quantifying, projecting and delivering on its commitments, which is often not the case. There may be intractable, systemic bugs that engineering cannot diagnose or solve, in which case all the QA forecasting in the world won’t make that problem disappear. Engineering may have written spaghetti code that they are afraid to touch, for fear of introducing
By Niall Lynch – email@example.com 310-829-2044
Adventures In Quality Assurance - page 84 of 86
more bugs caused by the fixes. They may have no grasp of how to deal with problems in emergent properties of the product, such as performance and security. Inherent conflicts in the product’s requirements may only have emerged at a late date in the project, and resolving them might require a whole new project. Etc. As I pointed out earlier, QA can’t change the quality of the requirements or the coding. But they can, potentially, know how good the code is and how resilient it is. Even in a situation where Engineering is unable to deliver a finished product, QA, in a quality assurance environment, will be able to know that, and know it fairly quickly. It can know this because it will be able to see that certain requirements just aren’t being validated, even though new iterations of code are being submitted to QA over and over again. This in turn will allow it to identify systemic code problems that are just being patched, not solved. Which means, in turn, that a properly constituted and run QA department has the ability to see that a project is “churning”, that is, trapped in a loop from which it cannot extricate itself, just as the “churn” is beginning. This is important, because one of the reasons so many projects are so very late is that the project team refuses to acknowledge they are trapped in such a loop, often for months at a time. Every week, the code will be ready the following week. This promise never changes as the weeks flow merrily by. The project is always one week away from being done, forever. Everyone in the project knows this is occurring, yet there is a strong psychological taboo against acknowledging it. Moreover, the perennial optimism of Engineers does not allow them, usually, to acknowledge they have reached an impasse. During the height
By Niall Lynch – firstname.lastname@example.org 310-829-2044
for determining when a project has entered Groundhog Day territory. concretely. I am happy to devote the second half of this book to a detailed exposition on how to apply this view of Quality Assurance. since there is nothing new to test.page 85 of 86 of project churn. and have shown how these principles would affect the most common problems of software development. then. since it is usually during the “churn” phase that a project team loses the confidence of upper management. and psychologically objective tool. I know in doing so I have left many detail questions unanswered. is the theoretical basis for my view of Quality Assurance. I hope. Moreover. is a key way that project teams can retain the confidence of upper management. I wanted to give you a chance to see my philosophy. what the basic principles of quality assurance really are (and are really not). avoiding such a phase. in all its parts and ramifications. I have skipped this level of detail because I did not want to get bogged down in operational minutia at the very beginning. Now that I have done that.Adventures In Quality Assurance . at a high level. describe what institutional and logistical realities they entail. however. with crystal clarity.com 310-829-2044 . but of the entire institutional and financial system that generates and supports them. bug counts may actually fall. This is why the ability to track validation of requirements provides a powerful. In Part II I will. flesh out how such principles would work in practice. and provide specific examples of By Niall Lynch – verlandosta@yahoo. a project will run on hope right off a cliff. and thus find a meaningful voice in ship decisions. to the work of software development. or exiting it as quickly as possible. even though validation rates are dismally low. This. I have outlined. I have tried to present a comprehensive analysis not just of quality assurance and its failings. Absent this data.
if you are not a QA person. and from hope to happiness.Adventures In Quality Assurance . Your plane flight. you may want to stop reading at this point.com 310-829-2044 . In any event. In other words. may be a short one. By Niall Lynch – verlandosta@yahoo. everyday problems of quality assurance. from principles to details. let us now turn our attention from ideas to realities.page 86 of 86 my theories applied to concrete. after all.
This action might not be possible to undo. Are you sure you want to continue?
We've moved you to where you read on your other device.
Get the full title to continue reading from where you left off, or restart the preview.