The Seven Stages of Expertise in Software Engineering

The Seven Stages of Expertise in Software Engineering
Meilir Page-Jones
© Wayland Systems Inc. All rights reserved. (In this article, an indefinite “he” means “he or she”.) For almost two decades we at Wayland Systems Inc. have toiled to transfer the latest developments in software engineering to practitioners at large and small companies around the world. However, until recently we didn’t do a very good job of quantifying the long-term field results of our efforts. So we decided to ask a question reminiscent of the rhetoric of Ed Koch (the former mayor of New York City): “How’re we doin’?” We surveyed some of our clients to whom we’d provided education and consulting over the years and discovered an interesting but very disturbing fact. Over 50% of the shops into which we’d introduced Software-Engineering techniques had either abandoned the use of the techniques or had let their band of active practitioners shrink to a dwindling core of diehards. What was going wrong here? We turned to self-examination and reviewed our courses and consulting practices. But, whatever the problem was, it didn’t seem limited to Wayland Systems: Informal discussions with our competitors revealed that they too were having similar problems. The attrition rate across our whole industry was painfully high. External examination revealed several factors at work, many of which were to do with corporate management habits. Examples of these were: unrealistic management expectations; a lack of commitment to quality; an obsession with coding (which we termed WIMP — “Why isn’t Marvin Programming?” — management); pernicious deadlines; and a mis-allocation of skills to tasks. Other factors included: engineers’ resistance to change; engineers’ reluctance to participate in team approaches; and the then-unavailabilty of useful CASE tools. The above factors were significant and worthy of articles in their own right. However, there was another factor, which is more important and fundamental than the others. This factor is the way that people are able to absorb sophisticated new techniques and then to apply them to their work — in other words the stages of expertise through which we all pass as we learn and use new and powerful disciplines. At first we were aware only that there were two types of person: novice and expert. That was hardly earth-shattering news. However, further research revealed that there are actually seven stages of expertise through which a person may pass on the journey from total ignorance to world-class knowledge. With devastating acumen we numbered these seven stages Stages 1 through 7. (We also gave them names: Innocent, Exposed, Apprentice, Practitioner, Journeyman, Master and Expert.) As I’ll discuss below, these seven stages have a deep impact on the successful — or otherwise — introduction of software engineering into an organization. A participant at a conference asked me how universal these seven stages are. “Very universal,” I answered.

1 of 7

3/28/2011 2:53 AM

Stage 1: Innocent A Stage 1 person has never seen or heard of bears. His manager is also keen to send him out with the latest bear-hunting techniques because the users want fur and they want it yesterday. in fact. Stage 3: Apprentice A Stage 3 has attended a 5-day seminar on bear hunting. Following that illustration I’ll outline the application of the Seven Stages to our own business of Software Engineering. if he encountered a bear. During this seminar. is full of confidence. 2 of 7 3/28/2011 2:53 AM . fierce bears. larger bears. It would not occur to a Stage 1. Unfortunately. In the heat of ursine confrontation. Stage 2: Exposed A Stage 2 person has seen an occasional bear and has read articles in airline magazines suggesting that bears may be hunted. Neither would he realize that a bear is a potential source of danger. the participants form into teams of three or four and practice hunting very small bears under the ever-watchful eye of the instructor. if you’ll bear with me.waysys. It’s typical that some Stage 4s get some bears. He is accurate and productive: The Steering Committee merely points out the bear and he hunts it within both budget and deadline. He is motivated to learn more. They fill out evaluation forms attesting that “bear hunting is very useful and relevant to my job. by Friday afternoon all the teams have successfully hunted their bears. a Stage 2 probably has friends that have hunted bears and has learned some desultory but intriguing facts about bears and their habits. “Yes. the Stage 4 may also forget or misinterpret his classroom instruction and precipitate disaster.html “You mean that I could even apply them to the skill of bear-hunting?” he retorted unexpectedly. I should like to illustrate the Seven Stages of Expertise with reference to hunting (although I’m most definitely not a fan of such an activity). Stage 5: Journeyman The Stage 5 has survived the traumas of Stage 4 and has bear hunting down cold. that the bear could be hunted. Moreover.The Seven Stages of Expertise in Software Engineering http://www. He’s ready to transcend the minuscule bears of the 5-day workshop and go out for real bears. having completed formal bear-hunting education. they are barely prepared for the world of real bears. but it’s also typical that some bears get some Stage 4s. Stage 4: Practitioner The Stage 4. The Stage 5 is the exemplary modern hunter that salespeople of bear-hunting seminars refer to in their brochures. After a few interim setbacks. he can’t imagine how he ever got along without them. The Stage 5 uses modern bear-hunting techniques naturally and automatically. in the resulting scramble the budding bear hunter may be sent out without a map and with the wrong caliber arrow in his longbow.” I said.” However. He’s ready for Ursa Major.

Now let’s come back to the world of software engineering and see how the Seven Stages of Expertise apply to us. Stage 6s know more than rules: They know why the rules exist and even when it’s permissible to break them. Stage 6s are very capable of training others in hunting techniques. Because of their deep knowledge. I’ve never tried the experiment and I don’t recommend that you do so either !) Many Stage 1s are experiencing “Frog in the Pan” and are trying to tackle problems of the 1990s with approaches of the 1960s and 1970s. a Stage 7 may extend bear-hunting to work also on Big Foot or he may even develop the ultimate Yeti-Oriented Hunting Methodology.waysys. You may find it incredible that Stage 1s could exist in the 1990s. a Stage 3 or 4 may stand upwind of a bear accidently and scare off the bear. it’s suddenly gotten hot in here! I think I should hop And the reason stems from the way in which software complexity evolved.html Stage 6: Master Stage 6 bear hunters have internalized not only the mechanics of bear hunting but also the principles underlying the techniques. without realizing that the problems they’re facing are the very ones that modern Software-Engineering techniques were created to alleviate.” This is because although a frog will jump out of a pan of hot water. They are also engaged in extending and generalizing bear-hunting techniques to solve new problems. Stage 1: Innocent A Stage 1 may not have heard of Software-Engineering techniques.” (Before I get into deeper trouble from animal-rights folks. However. Indeed. Software became insidiously more and more complex in the 1970s and 1980s as users demanded more and more sophisticated systems be installed on the more and more powerful hardware that became available. a Stage 6 may know that by wearing Yogi-Spray Deodorant he can stand upwind without being detected and can thus surprise the bear from an unexpected quarter.The Seven Stages of Expertise in Software Engineering http://www. Yet there was no sharp transition. For example. Stage 2: Exposed 3 of 7 3/28/2011 2:53 AM . For example. Stage 7: Researcher Stage 7s are asked to write books and give talks at bear-hunting user groups. Or — more likely nowadays — he may be vaguely aware of their existence but may not see their possible relevance to his situation. he may be only dimly aware that there are any software-development problems in his shop. The earth was not hit by a Complexity Asteroid in 1975 that suddenly made software three orders of magnitude more complex and cast our reptilian development techniques into extinction. but they do. a frog that is placed in a pan of cold water and slowly heated will fail to leap forth and will actually boil to death. I hasten to add that this analogy is apocryphal. The temperature gradient is so gradual that there will never be a point at which the frog declares: “Boy. I call the way in which software complexity actually increased “Frog in the Pan.

and on each successive project a Stage 5 further hones his skill and enhances his productivity. then it’s highly likely that the project itself will fail and the SoftwareEngineering techniques will be publicly pilloried and then abandoned. If a Stage 3 absorbs everything from a seminar. You could say that most Stage 3s know just enough to be dangerous! Stage 4: Practitioner The rite of passage to Stage 4 is the use of Software-Engineering techniques on at least one significant project. Stage 2s may survey magazines. full-sized project in the corporate jungle. deadlines and changing requirements. If an entire project is peopled with Stage 3s.html Stage 2s have noticed that the water is getting decidedly warm. Achieving “Stage 4-hood” is for many people the most difficult transition of the six transitions between stages. Their experience of Software-Engineering is “latched” in place and there is little risk of their reverting to the past. however. He continually needs consulting advice. then he is minimally equipped to tackle a true. In these workshops they tackled small but realistic case studies that resembled their own projects in miniature. A Stage 5 is self-sufficient .com/ws_content_al_sse. Stage 5: Journeyman Stage 5s have made it. the Stage 6 knows the “whys” of Software Engineering.or 100-fold.waysys. At the same time. a large proportion of Stage 3s never make it to Stage 4. Stage 3: Apprentice Stage 3s have attended one or two 5-day workshops on Software-Engineering techniques. Since many people throw up their hands at this point and revert to their old mediocre but familiar ways. Usually. In the Stage 5 the SoftwareEngineering techniques yield for the first time the productivity the marketing folks promised. The Stage 6 is 4 of 7 3/28/2011 2:53 AM . The case studies provided valuable kinesthetic reinforcement of the formal lecture material and were thus indispensable. However.The Seven Stages of Expertise in Software Engineering http://www. The fledgling Stage 4 is asked to take untried (by him) techniques and apply them to a corporate project with the usual demonic cocktail of politics. being limited to a few terms and definitions and not based on any practical Software-Engineering experience. without which he will encounter a series of minor setbacks or major failures. a Stage 3 does not grasp everything or has difficulty scaling the techniques up from a case study to a real project. he is attempting to recall what he learned in class and scale up the examples 10. So they are actively seeking Software-Engineering techniques that will get them out of the pan or at least reduce the heat. Stage 6: Master The Stage 6 not only is an adept technician.more often the source of Software-Engineering advice than its recipient. but also possesses a profound methodological foundation. while adhering to a more fundamental methodological principle. the case studies’ apparent realism conveyed to the Stage 3 a confidence that is often unwarranted. Beyond the “whats” and “hows”. if not downright hot. This depth allows him sometimes to break a surface rule. Their interest level is high but their knowledge level is low. confer with colleagues or attend one-day overviews of the techniques.

He also scans the horizon for new problems towards whose solution Software Engineering can be extended and generalized. Act now!” (By the way. The Stage 7 is at a methodological pinnacle. Depending on the individual person. Below I discuss four of these: The Productivity Curve.) Except for a few outstanding people. which represents the difficult rite of passage of the incipient Stage 4. this presentation (see Fig. A couple of our clients measured the length of time from Stages 3 to 5. The Productivity Curve Vendors who sell Software-Engineering techniques do so by convincing management of their boon to productivity.waysys. Stage 7: Researcher The Stage 7 is concerned with delivering the latest developments in Software Engineering to a wider audience. 1) is not a complete a good instructor because his theoretical and practical knowledge give him the wherewithal to tackle difficult student questions. via books. A. Fig. Pilot Projects 5 of 7 3/28/2011 2:53 AM . the shortest time from Software-Engineering seminar to exquisite software productivity is 24 months. Between Stage 3 and Stage 4 there’s a hair-raising “J-curve” of productivity (see Fig. However. Such a manager is not merely impatient. Although vendors rarely say so explicitly — we at Wayland Systems never used to — they compare the productivity of Stage 5s to that of Stage 2s. 2). no-one warned him of the J-curve and so when it manifests itself he can only assume that the much-touted Software-Engineering techniques were mere hokum. articles and conference appearances. When managers tell me: “We just can’t wait that long. (These times are very approximate because of the subjective nature of the stages. You may also decide what (if anything) you should do to progress to the next stage. You might think about a Software-Engineering technique that you know and consider which of the stages you are in with respect to that technique. This is because a portion of Stage 7s’ energy is directed away from the specific project and towards general methodological research. These Seven Stages of Expertise are valuable in their own right. whether you do anything or not. Unfortunately. 1 (to come) Fig. The Stage 7 looks out for flaws in contemporary Software-Engineering techniques and for consequent ways to improve the techniques.) B. notice that Stage 7s have lower project productivity than Stage 6s. Pilot Projects. there are organizational implications behind the Seven Stages. Stage 3 to Stage 4 took 6 to 18 months (about the length of a project or phase) and Stage 4 to Stage 5 took 18 to 36 months.” I retort: “The 2 years will pass anyway. 2 (to come) This J-curve has led more than one manager to abandon all hope in SoftwareEngineering techniques.The Seven Stages of Expertise in Software Engineering http://www. The Critical Consulting Core and Ephemeral Technology.

of course. If you don’t yet have such people on board.. But which of them with bone-crushing deadlines and ax-wielding users wants to be the first to experience the depths of the J-curve. some truth in this complaint. All agreed that this was a wonderful idea . Most project managers would agree that Software-Engineering techniques are a Good Thing. but instead continue to work on building up a pyramid of internal expertise with Stage 6s at the top that will ensure a lasting mastery of Software Engineering.waysys. although the delivered system is the most important product of most projects.html But the J-curve has a facet nastier than simple delay in productivity. The pilot should be short (6 months) and should yield a non-critical system. Core of Internal / External Consultants One of the surest recipes for disaster is to staff a project with only Stage 3s. The pilot will uncover all manner of pitfalls that occur during the Stage 3/4 transition as well as problems in the techniques themselves. People will still have to go from Stage 3 to Stage 4.” If you buy a power drill to replace a 6 of 7 3/28/2011 2:53 AM . treat every project as a pilot.) Every project should have access to a Stage 5. or . In that sense. A pilot project will not eliminate J-curves on subsequent projects. Yet many managers do exactly that. a group of mice who’d been persecuted by a cat decided to hang a bell around the cat’s neck that would warn them of its approach. plus c’est la même chose. then I recommend that you try SoftwareEngineering techniques on a pilot project first. you should never forget the learning by-product of every project. In the fable of that title. If “who will bell the cat?” is a problem.. but there was one minor difficulty: None of the mice wanted to be the one to risk walking up to the sleeping cat to tie a bell around its neck. pushing a group of analysts through a seminar one week and letting them loose on a crucial project the next week. This is the “Who will bell the cat?” problem. Cultivate your own Stage 5s and Stage 6s in-house as part of your long-term staff-development policy and make them available to project teams. don’t become chronically reliant on external consultants. that technique had been left behind in the mud of obsolescence by the relentless march of progress. However. (Me neither!) Thus many shopfuls of intelligent people appear to be dinosaurs because the Technology Trap holds them in the 1970s. Although there is. (Jerry Weinberg refers to this as the “Sheep Dip” approach. then call in external consultants from Wayland Systems or one of our competitors.better still — have a Stage 5 on the project This impasse is known formally as the Technology Trap. this is one project where the learning experience is more important than the end-product. it is also true that “plus ça change.The Seven Stages of Expertise in Software Engineering http://www. You should learn as much as you can about applying the new technology from the pilot project. D. Review what you discovered about doing projects when you complete each project and share that information with others. C. Learning Latency vs Lifetime of Technique One manager of a major organization recently complained to me that by the time a critical mass of his people had learned a Software-Engineering technique. Indeed. as some causes of the curve’s nadir are removed. But the pilot will make future J-curves more shallow.

because many of the new ways of addressing software are simply refinements of ways already known to Stages 3. . Have each person state their medium. There will be much to learn — even for Stage 6s and Stage 7s — as we pass into the new If you don’t have a Stage 5.html manual drill. Similarly. Superimpose technology on to the strategy in order to yield tactics. These people will be able quickly to absorb a new technique and to assess its applicability to the shop’s needs. the mission of Wayland Systems is “Mastery in Software Engineering. Establish similar medium. beg. This mission will not evaporate for many decades yet. Determine the most appropriate blend of stages for the work that you do and set up a plan to achieve that blend. For example. borrow. kidnap... Finally — Some Recommendations Be aware of the Seven Stages of Expertise and their effects on productivity under a specific Software-Engineering technique. They will also be able to compare and contrast the new technique to current techniques and to steady the transition from old to new. Stabilize the effects of technological change by nurturing an in-house expertise (Stage 6) in fundamental methodological principles. Seed each project with Stage 4s and allow access to Stage 5s and 6s if possible. 4 and 5.The Seven Stages of Expertise in Software Engineering http://www.” In other words. Try to formulate shop strategies (as much as you can) in non-technological terms. steal.waysys. for software is becoming ever more complex and ever more critical. Have each person assess his stage with respect to each Software-Engineering technique that your shop uses. External consultants are valuable as a short-term measure. Develop a large “capacitance” of Stage 6s in your organization in order to smooth the apparent high-frequency ripples in technology. we seek to assist our clients in becoming self-sustaining at at least the Stage 6 level of Software Engineering. 7 of 7 3/28/2011 2:53 AM . Take stock of where you are. Never attempt a crucial project solely with Stage 3s and below. Acknowlegments I should like to thank my colleague Claude Maley for inspiring me in the first place to go out and search for the Seven Stages at our client sites and my colleague Steven Weiss for helping me to name and define the characteristics of each of the Seven Stages. Reconcile your expectations for Software Engineering with the realities of the Seven Stages.and long-term goals for the shop as whole.and long-term goals for expertise in each relevant technique and suggest a plan for achieving those goals. Have a mission that will never go away. buy. most new Software-Engineering methods are basically founded on familiar underlying methodological principles (such as the principle of “Safe Deferral” or of “Partitioning to Minimize Interfaces”). the deep principles of drilling will be largely unchanged. one.

Sign up to vote on this title
UsefulNot useful