This action might not be possible to undo. Are you sure you want to continue?
Where Do We Go From Here?
Analytical Tools to Help Leaders Navigate Uncertainty and Risk
by Mark Jones Jr.
© 2011 by Mark Jones Jr. Published by Rose Petal Press, the publishing division of Mathematical Christian Consulting (mc2), www.multiplyleadership.com. ISBN: 978-0-9826572-1-8 Library of Congress Control Number: 2011938693
.For Beth – “A virtuous woman is a crown to her husband” (Proverbs 12:4a).
....................................................... 15 A Picture is Worth 10.......................................................... 30 Applying the ATOMs – Another Example ............... 9 Chart the Course: An Organizational Construct ...................................................................................................................................... 18 You aren’t Average .................................................................................................................................................................................................................................................................................................................................................................................... 21 Conclusion .....................................................................................................................................Foreword .......................................000 Words ........................ 28 Appendix – Supplemental Materials .............................................................................. 6 Introduction ...................................................................................................... 16 Two Wrongs DO Make a Right .............................................. 25 Afterword .............. 33 ................... 29 A Low-‐Level Journey – Another Chart the Course Example ...................................... 7 A Cross-‐Country Journey ...................................................................................................... 11 Three ATOMs ........................................................................................................................................
The path ahead is clear. the picture on the cover is blurry.Foreword Yes. . That was intentional. It doesn’t have to stay blurry. Demonstrating this to you is the purpose of this white paper manifesto.
or did not brief it at all. This resulted from a change in the airspeed parameters for the test. failed to ensure that a proper recovery was flown. refusing to alter the status quo. we won't see him or her in time. The pilot. the kind that makes you want to stomp on the brakes—just hard enough to stop as quickly as possible but not so hard that you hydroplane. We are afraid to keep going.Introduction We are drowning in data. don’t want to be on the receiving end of a crashing vehicle behind us that decided not to stop. The first piece of information lost in the deluge was a minor change in the altitude of the maneuver. The chase plane was scanning for other air traffic. at least three parties. A test point much like this had already been performed. lost because there was not enough altitude to recover. Big mistakes are not the problem—it is the cumulative effect of a million little things. because the visibility is so poor we can't even see out the front window. We were preoccupied with a million little things. because of everything we just thought—we don't want to be rear-ended. In any case. . It was a flight test technique that we understood well—even students at the test pilot school practiced a similar maneuver. In times like these. our default response is often to stop making decisions. or we'll skid on the slippery pavement and come to a crashing halt. the test pilot. stopped. letting our momentum keep us on the course we had previously decided upon before the wall of water came gushing from the sky. Consider the March 2009 crash of an F-22 in the desert just north of Edwards Air Force Base. If someone is there. who ejected. probably many times. It is possible that those who reviewed the test and safety plan also neglected this critical piece of data. a chase plane. I believe that only the pilot tracked this data and only tracked it during execution. Most everybody knows that you don't do a supersonic split-S. The airplane's remains were nothing more than a smoking hole in the California desert. But nobody was thinking about it that day. Another piece of information swept away by the hustle and bustle was the type of recovery to be flown. The airplane was lost in the bottom half of a split-S maneuver performed at one-and-a-half times the speed of sound. and the control room personnel stopped collecting data after achieving the test point’s target parameters. Until it was too late. It is unclear to me whether the test team briefed the recovery maneuver correctly but did not execute it—briefed it incorrectly. It's a torrential downpour. It was lower. We are afraid to stop. It's a flood of biblical proportions. This kind of indecision is fatal. succumbed almost immediately to the injuries sustained from bailing out at a speed where the relative wind is like a brick wall. and a ground-based test conductor in a control room.
at the end of the Union line. was critical. We are surrounded. See You at the Top. and we are surrounded by scarcity. LA: Pelican. His 300 men had dwindled to 80.After the accident. we are faced with increasingly scarce resources—fewer airplanes. Zig. The enemy was north. It had been around for years. for the first time in the history of this campaign. Do it faster. These enemies are far too easy to picture. It was another fact that had been lost in the deluge of data. runways. 2000. to say the least. Our choice is the same—to advance or do nothing. The situation is dire.’1 What is Our Plan of attack? Almost a century before that. we are now in a position to attack the enemy in any direction. the armored artillery that changed modern war. What is our weapon? It’s not data. His reaction to this news. They could not fail. Joshua Lawrence Chamberlain stood on the peak of Little Round Top. On another side. General Creighton Abrams and his command were totally surrounded. These resource constraints also create a tremendous burden on our schedules. Sigma. In every aspect of our enterprise. But it wasn’t new or novel. We have tools more effective than General Abrams’ tanks. He faced an uncertain future—what would he do? The Rebel yell sounding from the bottom of the hill forced him to a decision point. a special team developed a new test planning process—an algorithmic approach to dive recovery planning called time safety margin. and west. We must advance. don’t shoulder the burden of responsibility. south. Costs are rising while budgets shrink. Customers are stingier with spending. and shrinking airspace—these things creating congestion and resulting in hazardous traffic conditions. At one point during World War II. But their position. east. On one side. Do more. We face uncertainty in a flood of data. They had exhausted their supply of ammunition. ‘Gentlemen. mu. Indecision or action. . 1 Ziglar. This isn’t the first time in history that we’ve been in dire straits. he had a choice to make. Gretna. We have the ultimate high ground from which to mount our charge. And do it cheaper. alpha. and beta. a paradoxical scarcity of insight in a seemingly endless ocean of information. Budget completes the circle of enemies surrounding us. we have this deluge of data. We don’t want statisticians spouting coefficients and variance and telling us about logarithmic regression. But we have something more powerful than the cold steel of the bayonet. we face the ever present demands of schedule—always being driven to do more things in less time. Research money is scarce. and every other Greek letter.
add followers. Tools to Build Unfortunately. We want to understand and quantify the risk. That’s our weapon. a team—looking to us for leadership. Do you understand the inference? Notice that as I painted a picture of the situation we were in and listed the constraints. you’ll need to come with me on a journey not unlike one that you have probably flown many times. My goal is not to equip you with new tools. Leaders do. It’s about the artist. I did not say that manpower was scarce. You are a leader. Not only do I intend to show that you are already equipped but also that you are already a builder and an artist. our greatest asset. It is my goal to illuminate the subject and commission you to paint it. You have already been building. Instead. and to harness its energy—you have the people to accomplish it. how to use your momentum—your high ground—to your advantage. the strengths and experiences you already have. and painted the background colors. to focus its power. It’s your job to inspire them for the task at hand. John Maxwell said. It’s not about the tools. We couldn’t pick it out from a lineup of paintbrushes. I want to show you that you are already equipped with the fundamental tools you need. but to multiply growth. And we don’t have enough people familiar with it to oversee and assist those who don’t know it.We want enough information to make the right decision. I’ll show you what strengths you have for the task at hand and how to leverage those strengths. did not say it was one of the obstacles we faced. We don’t have time to make sure every engineer attends a course on design of experiments. “To add growth. add leaders. I also suspect that you and I wouldn’t be able to identify Michelangelo’s paintbrush. Leaders. because pencil pushers don’t make decisions. having tools doesn’t make you a builder any more than memorizing the dictionary doesn’t make you a writer. To show you these things. I want you to build a dam to tame the flood of data. and leadership is a way to multiply the effectiveness of this. laid the foundation. We must lead.” We have surrounding us a work force. We don’t have the time to teach a team of mathematicians everything they need to know about aerospace. a team of tremendous potential. to guide our judgment and reinforce our gut. and flight-testing. That’s okay. We have people—followers. It’s about leaders and leadership. A Cross-Country Journey . That’s what we’ll build on. engineering. We don’t have the time to become a statistician.
I should pass over the southern tip of a large pond. After five minutes on a 255 degree heading. I thought . Once I did. because I wanted to make sure I had enough fuel. the closest airport with a control tower. I opened up the sectional charts and sat down to figure out where I wanted to go. But that wasn’t part of my cross-country training. Learning to fly cross-country was probably one of the easiest parts of the training—or that’s what I thought back then. I should cross over a major highway with an overpass to my left. but of course. I could jot down a heading on my kneeboard or on the chart and just remember that. My instructor’s name was Ian. The pond is just north of my position. In fact. It was inside class D airspace. “Can ducks make vertical turns with turbulence”—that’s what Ian told me. the winds are drifting me south of my intended course. Deviation. Those two things would get me pretty close: 255 degrees magnetic for 55 minutes. and True heading. I correct my heading to 260. Ground speed and fuel burn weren’t included in the duck mnemonic. Variance. At 15 minutes. I was high and hot on final. so I didn’t want to wander in without contacting the tower. It’s not as far south as I thought it would be. True course. a pilot’s head is full of crazy sayings and silly words that mean something when translated into aviation jargon. I look outside and see where it is. he predicted I would never forget it before he even told me. At seven-and-a-half minutes. The fact is. where I lived in a two-bedroom apartment. I remember the first time I went to an airport that I had never visited. so I managed to land. I ran through all the steps every time. Several times. Magnetic course. Doing what I “knew” how to do was harder than it looked. I should overfly an intersection in a small town. Back at Cook County Airport. and the place was Cook County Airport (K15J) in southern Georgia. and I was in a little airplane.Remember back to the days when you were learning how to fly? For me. My performance revealed otherwise. I am just north of the intersection. tail number four-hotel-bravo. That should keep me on course. and I can tell you from experience that navigating to my destination became challenging. When I walked into the flight school that day. Two-five-seven is right in between. Apparently. Wind correction. This mnemonic helps me remember the steps needed to plan a cross-country flight. Remembering a wacky sentence about ducks is easier than remembering Compass heading. the plane was a Cessna 150. A heading of 260 degrees corrected me back to course and then a bit right. they were important. just a place where I learned another lesson. just north of Valdosta. He flew seaplanes somewhere in the South Pacific for many years before teaching private pilot students. But it was a long runway. Sometimes I would jot down a planned duration. I didn’t plan at that level of detail. But I didn’t want to contact the tower until I had found the airport. The first time I went alone to Valdosta Regional Airport (KVLD). Ian told me something I would never forget.
These stories should also begin to suggest that you can extend the application of what you already know about navigation and uncertainty. we will return to it often. based on the ground references I saw out the window. developing even more complex and elegant applications of those tools. The airport was hard to find. but it was much later when I learned the “clock-to-map-to-ground” tool and added it to my bag of navigation tricks. I found it eventually. and validate. 2 . I multiplied my navigation capability tremendously. an anecdote upon which we will lay the foundation for the application of the three most important and fundamental tools for taming the deluge of data and transforming the uncertainty into decisions. algorithm.2 I keep building on that initial training. I still haven’t mastered those simple techniques—don’t let anyone tell you they have. decisions and judgment. The stories above—and the chart the course process—will serve as a map. Pilots of every experience level and the most senior executives alike have used it.I knew where I was. applying these fundamental tools. execute. on night vision goggles to blacked out drop zones. more importantly. For that reason. Getting Things Done. its details will help you recollect the three ATOMs and their application. illustration. and process for understanding and. By intentionally adding the chart to my composite crosscheck in a methodical way. Chart the Course: An Organizational Construct I wasn’t done learning this tool embodied in the words charting the course. Even in the precise airdrop strikes accomplished at night. repeatable process for planning that is compatible with the scientific method. Perhaps you will tire of it now. it is now in the appendix. The three steps to this process are as follows: 1) Where am I going? 2) Where am I now? 3) Where are the waypoints? The original version included an additional illustration from C-17 special operations low level navigation. Charting the course consists of three steps: Plan. It is also analogous to the methodology of managerial expert David Allen in his landmark treatise. (Though I will not discuss his interpretation here. I mention it for your reference). but I didn’t. we still hadn’t mastered the tools. Imagining the cross-country flight from our early student pilot days provides us with a vivid mental image that will serve us well in our methodical study of ATOMs (analytical tools of mathematics and statistics). into action. Plan The first thing this story provides is a scalable. Chart the Course is simultaneously an algorithm and organizational construct. heuristics and analytical tools. but in the future.
and the problems we face impose restraints like those illustrated in our example. Where are the waypoints? The route may not be a straight line. selecting waypoints based on distinguishing features is appropriate and desirable. the outcome of charting the course. purpose. From the origin. mission. Many pilots take off. practice takeoffs and landings and touch-and-gos. There may be obstacles. or the boundaries of a current scientific theory. This process is mission based. fly their practice maneuvers within sight of the airport. Our organization. or reason. For example. However. our team. we use great circle routes. there are 360 (whole number) directions from which to choose. we might define our destination as a point in time. in which case. We know that two points define a line—and that line. That would be wrong. goal. the plan for getting there—is the goal of this planning process. whatever metrics we use to measure where we are presently must also describe our destination. Some journeys may even require stopover points for refueling or overnight rest. Perhaps this is a commentary on human imagination. The process is cyclic in this regard. It assumes you have a vision. This process will get you to your why. or airspace that impose limitations on the plan. the laws of nature impose certain limitations on the destinations we may choose. and yet never depart. you must find your destination on the chart. Therefore. Execute . In the analogy. whether that language is the symbols of the aeronautical chart. This step is critical. Or one may choose none at all. This is not the intent of this process. as we consider our present location. we should describe our origin using time as a primary characteristic. We navigate to places we can describe using the language we know. This is no different from what we encounter in the realities of flight test and scientific experimentation. if geography is a significant descriptor of where we are. In navigating very long distances. Where am I now? Using language common to that used to identify our destination.Where am I going? You might think that we should start with the point of departure. we must define our present location. than we should include that in a description of our destination. In flight test. one may choose a route based on the ability to navigate accurately. Additionally. However. the limits of human understanding. the limitations of the aircraft equipment and range may affect these decisions as well. terrain. However.
but the recurrence of this chart the course theme is of the utmost importance. Notice that we do not look behind us. Now begins the scalable. In the performance phase. We are established on the desired 255 degree heading at the appropriate airspeed and altitude. It answers the where am I going question and clearly describes how we will know when we have arrived. we should. and beside the airplane to verify our position. So let us continue with the assumption that the desired attitude has been set. The aircraft accelerates down the runway and gracefully enters into the most sublime of human activities.. Our ability to steer a course is not without variation. . The clock and these landmarks depicted on the chart are targets for our metrics. We set the attitude that achieves the desired performance. What was our destination? Was it a given climb rate? Where did we start? Perhaps we started from level flight or already had a climb established but not at the desired rate. In this next phase. “At five minutes on that heading. ahead of. we find vestiges of the planning process. But even in this control and performance method of execution. powered flight. At this stage. heading. We have planned and executed. using our instruments. we verify. the work of the leader begins—the judgment acquired only by practice and experience—and the pilot earns his pay. at least not in an airplane. to confirm the fidelity of our execution. though the steps are almost identical to those we have already encountered. In flight.With a course decided upon.. The answers to these questions affect the magnitude of the change and so define the waypoints. now the intrepid student pilot from our example above must execute the plan so carefully constructed. repeatable process of execution. and now we verify or make course corrections as necessary. that the attitude established had the desired effect. We determine the attitude that is desired (where I am going). And we make the inputs (waypoints) necessary to effect the change. We search the ground outside. We ascertain our current attitude (where I am now). this requires us to control altitude. This is similar to the noise that we find present in flight test data or in any scientific endeavor. We must accept this reality. These steps are the control phase of the control and performance process. and airspeed. we describe the process with the phrase clock-map-ground. Turbulence bounces the nose of the aircraft around the sky.” This information is extracted from the chart—from the plan. Validate Course corrections are an integral part of the pilot’s life—the wind is never as planned. I do not want to labor these points.
But even in that early stage. will demonstrate.Anyone can plan this cross-country flight. The point or origin is the composition of the test team and the nature of the test article. Testing is the very essence of what we do. or objective. and validation is the follow-up procedures and processes we employ to certify aircraft and evaluate their designs. decisive action. and exercises the leadership required. “Predict-test-validate” is a phrase that I hope is familiar to the flight test community and especially to its leaders. . doing nothing was like steering a ship. or the flight test. The waypoints are the limitations that the nature of the problem. execution. So what if the aircraft is not on course? How great is the deviation? The skilled pilot will not drive the deviation down to zero but will accept some bounded quantity of error. the expertise of the pilot is what allows for the interpretation of the chart and the application of judgment in choice of route. That is the domain of the technicians. The destination is the assertion that we believe the experiment. At some point. Many variables affect this decision. the pilot will make a qualitative or quantitative decision that performance does not meet expectation and adjust accordingly. that’s leadership. the expertise comes to bear upon the decision of what corrective inputs to make. we must have a vision. however.Test . but the analogy allows us to apply it directly to flight test and the scientific method. In order to lead. we have described the planning. mission. But it is in the phase of validation—course correction—that a pilot becomes more than a passenger. There is a big difference between steering the ship and charting the course. But it also serves to emphasize the importance of vision and purpose. and validation phases of charting the course. flooding the road and obscuring our sight. or budget impose. Introducing this methodology of charting the course is a continuation of the illustration and analogy prevalent throughout. we tolerate a few feet above or below altitude. Predict . This actually reduces the workload of the pilot and creates a smoother ride and easier navigation. To a certain degree.Validate Up to this point. where he or she assumes the responsibility imposed by the title pilot-in-command or aircraft commander or captain. Even when this adjustment is made. resources. Planning is equivalent to the predict stage. A few feet left or right of course is inconsequential. It wasn’t leadership. Making an intentional decision to continue or to stop. Insisting that the first step of this charting process be identification of a destination has an important purpose in a grander scheme. When the rains falls in sheets.
I want to help you change your perspective. for those who find themselves in situations like Chamberlain or Abrams. Similarly. not defeat. This is a call to decisive action. Drawing the sword. to transform it into something useful—that is our purpose. but opportunity. This rest of this paper serves as an introduction to these ATOMs. I will show you three fundamental tools that you already know how to use. I did not write this for those who only steer ships. ATOMs are the fundamental building blocks of the most advanced mathematical techniques and graphical tools and data analyses. tools that you have already begun to master in a simple task like charting a course and navigating across the country. you will find three things: . and charging the mountain like Joshua Chamberlain—that’s decisive action—that’s leadership. It is for leaders and for those who aspire to lead. transforming data into decisive action.Examining the situation and framing it like General Abrams did—choosing to have that kind of visionary perspective—that’s leadership. In nature. His tools only took a minute to learn—but a lifetime to master. I want us to associate the word ATOMs with these tools in the same way that the term FTT indicates Flight Test Technique. to see like General Abrams. We are drowning in data. but do not let that cause complacency. We need a dam to harness the power of the flood of data. We already have the tools and the raw materials. ATOMs are the fundamental building blocks that a leader must use to transform data into decision. in the traditional meaning of the word. building blocks of molecules and collections of molecules that make up materials that are more complex. and risk. atoms. You and I can pick up a paintbrush and create a recognizable image on a canvas. We will call these three tools ATOMs. I want you to associate ATOM with elementary but not with simple. uncertainty. These tools may only take you a few minutes to read. are elementary particles. Three ATOMs Allow me to introduce you to generalizations of three analytical tools that you already fundamentally understand in a very specific context of dead reckoning and pilotage. like doublets and windup turns and roller coasters. It’s an acronym that stands for Analytical Tools of Mathematics and statistics. when wielded by the masterful hands of Michelangelo. But it’s also another word picture. FTTs are the fundamental building blocks of flight test maneuvers. Even a child understands how to finger paint. Just as atoms are the fundamental building blocks of nature. But the very same paintbrush. played a part in creating the wondrous ceiling of the Sistine Chapel. With each one. I will help you see what they are. and we will discuss how to use them to navigate uncertainty and risk. ordering men to fix bayonets.
A description of the tool. Compare what you see on the horizon to what you see on your primary flight instruments. to what you see on the attitude indicator. the first analytical tool of mathematics and statistics. A prescription for how to use the tool effectively. . of reasoning. I would suggest that a picture is worth 10. should tell the whole story. That is the core of this simple truth. What do you see? A trillion pieces of information. The story of the cross-country flight is the perfect illustration of this very important principle. The trend line is conclusion. We can interpret the world we see and understand it. at the sky above and the world below stretching farther than you can accurately imagine.000 Words “A picture is worth 10. when you compare your intended path on the chart to your actual track across the ground. fundamentally (so you understand its core principle) and creatively (so you won’t forget it). It is not. It is life and leadership. more data than you can fathom or quantify.1. You see evidence. With this in mind. A picture is worth 10. Compare your discernment of the attitude of the airplane. not evidence. and 3. It is far more complex. because I want you to read them quickly and often. You do not see conclusions. in the context of Charting the Course. I do not want to offer so many suggestions or explanations that you cannot see creative applications of these tools and learn to master them in a way that best suits your leadership style and the needs of your organization. and use them correctly.000 words. its application. A demonstration of the tool. But the applications are far-reaching and subtle and require great wisdom in implementation. What you see outside has incomparably better resolution. as much as any viewer would expect to see. we understand how to view data and draw conclusions. 2. nor should it be. 10. The following sections are brief. we should use these visual data in two ways.000 words of conclusion.000 words of data. the projection of the horizon on the windscreen. Look outside the airplane. do not see a sign that says. you gather clues that suggest whether or not you are on course.” The leader must use charts and graphics to visually represent and interpret data. A Picture is Worth 10. See Data not Conclusions First. Do you understand the difference? Points on a scatter plot should not be connected.000 words—this is the fundamental and creative description of the first of the three ATOMs. “You are here. I want each section to stand alone yet fit perfectly within the organizational construct and in the grander scheme. to confirm what we believe and to find our way when we are lost. In the chart the course context. of information. So a chart or illustration should provide evidence.” This isn’t a shopping mall. From the cockpit.
Begin with an assumption. As a common example of this process in action. If it’s to the right of your nose. and then look to see evidence that would follow if it were true. you begin to wonder if your drift angle is wrong or perhaps that you’re not steering consistently. you look just to the left of your aircraft’s nose. we rely on an assumption of normality. something that will confirm the position of the lake. and then 2. you look on the aeronautical chart and you see a lake just to the left of your course line at a given time. We plot these residuals on a normal probability plot and imagine that a fat pencil is placed over the points and the straight line representing the normal distribution. assuming that we should be and believing that we are. Significant departures from what you expect will begin to suggest that perhaps you were wrong. This is an important distinction. Instead. you rest assured that you are on course. We force the evidence to disprove our assumption. This is the same subtlety found in the logical foundations of the hypothesis test. Looking forward through the windscreen. examining road . and suddenly you don’t recognize what you see. but its inverse is appropriate. we assume that the residuals are distributed normally. we continue with our analysis. If these points are now hidden. If the lake is there where you expect it. We begin with an answer to the question. Looking again at your chart. begin to consider the alternative that our assumption is wrong. There may come a time when the evidence that you expected is not present. Find our Way When Lost The second use of visual data is for finding our way when we are lost. consider the illustration known as the fat pencil test.Confirm What We Believe Using the clock-map-ground technique as pilots. At this point. you begin to gather clues. satisfied that our assumption is true. This is exactly the process we follow when we “predict” and then “test” and then “validate”. You already believe that it is true. As a student pilot. we imagine a slightly fatter one. you search for another piece of evidence. If however. we begin to listen to the argument this chart is making. we correlate the time on the clock to a position on the ground. Perhaps you missed your exit. In using visual data. “where am I” and then compare reality to the chart to validate our belief. Know what you are looking for. this is the process: 1. In ANOVA (analysis of variance). then we smugly accept what we had already assumed. If the imaginary pencil covers the points. The clock-map-ground technique is not particularly suited for this task. at a given point in space. When still more points remain uncovered by our slightly fatter pencil. one or two points stray from beneath our nominal fat pencil. You begin outside the vehicle. You are not looking for proof of an assertion’s veracity. you are expecting to see the evidence that is present when it is true. Look for it. In many statistical techniques. Driving through an unfamiliar town is a pertinent example of this. Or you thought you knew where you were going.
Before we explain this principle. 2. In the formal study of logic. in both prescriptions for how to use pictures. charting the course. This ATOM is vitally important. But in those situations when it does occur. core beliefs. used to verify what we already believe and help us find our way when lost. and graphics. The student pilot then looks down at the chart in his lap and finds a road that looks remarkably similar near the penciled course line and concludes that he or she is on course. 3. By gathering sufficient clues. You ignore the color of the asphalt. There is Smith Street.) Why do two wrongs make a right? The answer lies in some rather obscure rules of logic. you apply these clues to the map. The pilot accomplished the steps in the wrong order.000 words of data and evidence. and you know you just passed it. prior knowledge. You look outside again to see Smith Street and apply this to the map. But I beg for your patience and persistence. you have answered the question of location. Note also what data we gathered. descriptive statistics help to sharpen the focus and illuminate the nature of the problem. You probably will not understand it after reading it the first time. Rarely does the engineer or mathematician enter a situation or encounter a problem without some underlying knowledge or core beliefs. not conclusion. Suppose you find Jones Street on the map.signs. and a variation of this technique helps us when we are lost. And thus. and from these two pieces of information can derive a plan to correct your situation. . the phase of the moon. are a fundamental element. most of the information that you could gather is irrelevant. I cannot emphasize enough. I want to illustrate how it could happen. assumption. and the beauty of the sunset. You are found. Description: A picture is worth 10. the waypoints between here and there. You now know where you are. Prescription: A picture is worth 10. Two Wrongs DO Make a Right (This ATOM is difficult. perhaps looking to ascertain the direction of travel by natural clues. Example application: “Clock-map-ground” illustrates how we use charts to compare predicted and actual performance. we determine the truth value of a statement using a strict set of rules. attempting to find the street names you’ve observed. and in reality. the process begins again. charts. It is so important. You probably did not even realize that you were filtering the information.000 words – data visualization. ATOM #1 Summary 1. This is the role of descriptive statistics. Now you have two points on the map and understand the direction you are traveling. that it’s the second of the three ATOMs. “Where were we going?” in this case it was your intended route of travel. Since the pilot is a student flying solo cross-country. he looks outside and sees a road. Once gathered. Imagine the cockpit of a Cessna 152 such as a student pilot might fly.
“Is the aircraft will accelerate a true statement?” to determine the overall truth value of the statement. In other words FALSE implies FALSE is true overall. Suppose we have an aircraft in straight and level. we also expect that the aircraft is not accelerating. please see this informative explanation and its supporting documentation: http://en. we cannot disprove preposterous statements like “if the airplane flies backward. an obscure rule of logic allows that if the former and the latter statements are both false. It is false—thrust must equal drag in straight and level.wikipedia. thus allowing us to make sense of the whole statement. In fact. unaccelerated flight. as we would expect. the latter phrase. Then we know that “thrust greater than drag” is not true. we know that this holds true and can demonstrate using logical steps and mathematical rigor that it should be so. unaccelerated flight.” is also false. when both sub-statements were false. then the airplane accelerates. Airplanes do not fly backwards.3 This is simply a rule of logic we must accept. Logic evaluates the truth of each sub-statement. asking the independent questions “is thrust greater than drag a true statement?'' And simultaneously but independently it asks. As previously stated. “aircraft will accelerate. It makes perfect sense that both pieces of the conditional phrase are independently false in straight and level. According to the rules of logic.org/wiki/Vacuous_truth. With that example as our motivation. In this case. check rides have been busted because of similar navigational errors and confirmation biases. the third case is the most obscure: when the first part of the conditional phrase (the phrase following the “if”) is false. In the first case above. then the entire conditional statement is true regardless of the second phrase. The laws of logic are precisely correct. the rules of logic add another layer of subtlety (and perhaps confusion for the reader). One such statement we might evaluate to ascertain truth value is the conditional statement. then the overall truth value of the statement is true. And we have all probably experienced this mistake long ago in our beginning navigation lessons." In the simplified case of straight and level flight. and in just a moment. In other words. I will illustrate how we fall in its trap frequently. Let's briefly examine why this makes sense. more commonly known as an if-then statement. the conditional statement as a whole was true. we must begin by imagining a situation in which the airplane is flying backwards (and falling straight backward to earth at the top of a vertical climb is not “flying backwards”). when both substatements are true. A simple example of such a statement is this “if thrust exceeds drag then an aircraft will accelerate. If For additional references. let us begin our explanation with an example.” To disprove it. and FALSE implies TRUE is true overall.convinced that he was somewhere when in fact he was not. However. The conditional statement is also true. 3 . Because of this rule. unaccelerated flight. we must accept this if-then statement as true. We cannot imagine such an instance. However.
Why do we. Here are just three examples of common assumptions. In fact. and prior beliefs and. and it relies on several assumptions. Let me expand an earlier example.. For example. You have a set of beliefs and assumptions and hypotheses. When you are navigating you use assumptions and judgment and prior knowledge. I want to illustrate how we often fall prey to this mistake. exist even outside of abstract statistical methods. and it is even more intuitive than normal statistics. and to use another word. and wisdom. I am not suggesting for a moment that we should not assume. If we fail to check that our assumptions hold. You get paid to use your judgment and prior knowledge—don’t abandon it when you start to talk about data and statistics. two that are statistical and one that is commonly found in aeronautical engineering: Normality of residuals in ANOVA. Thus. and I will show you an airplane that is accelerating. One of the common statistical techniques we use is known as ANOVA (analysis of variance). experience. assumption of the null hypothesis. if we start the reduction of data or the dissection of a problem from an incorrect premise—that is.two pilots argued over the matter. The human mind is still capable of far more than we even understand. We use this technique to determine if different factors (like configuration of an aircraft) have different effects (on performance. The p-value is meaningless if the setup of the experiment failed to satisfy the assumptions of the ANOVA model. There is an entire body of statistics that uses your prior knowledge and judgment and expertise. Assumptions. “But the p-value is so small. our assumptions. and it is this quality that I commend. experience. for example). how many times do you need to witness a destructively divergent PIO before you admit it is a problem? How many times do you need to flip a penny to convince yourself that the probability of heads is fifty percent? (Don’t even think about using the words confidence or power in your answer!) How many times does a plane need to crash.” you say. ignore what we know when we start to talk about statistics? Because we don’t know enough about ATOMs to realize that what we know is vitally important. I espouse the exact opposite. Use judgment. an entire branch of mathematics known as Bayesian statistics highlights the importance of our judgment. If this . even in the most general case. ultimately. we might make an erroneous conclusion. early and often. Do you need statistical design of experiments to demonstrate that a penny is fair and will result with equal probability in heads or tails? No. Assume. our initial assumption is false according to the laws of logic—we can arrive at a conclusion that is also false without violating any rules en route to that conclusion. and incompressible fluids below certain Mach numbers. You have knowledge and experience and judgment. as flight test professionals.” Obscure rules of logic aside. Our assumptions are very powerful.. beliefs. one might say. This is problematic because assumptions abound. “You show me an airplane that is flying backwards.
Use the Right Statistic You are continually cross checking your attitude and heading and track over the ground to make sure that it matches your plan. ATOM #2 Summary 1. 3. There is nothing average about what you do. You aren’t average. Prescription: Begin with prior knowledge and experience. After such a rigorous examination. weight and balance. and do this before we draw conclusions. Our engineering judgment is a key element of our leadership. you will rely on your prior beliefs. Example application: 1) The student pilot looks first at the ground and sees a road. models and simulations. the central tendency. Our predictions are critical parts of our test and safety plans. Stop leaning on the crutches of confidence and power. because you weren’t average. then—in the wrong order—looks at his chart to convince himself that he knows where he is. to achieve the performance as depicted on your chart. Description: Two wrongs DO make a right – assumptions and prior knowledge. And you didn’t conform to what everyone else thought or did. You are in the position that you are.were a flight test. . because you have prior knowledge. with equal probability of heads and tails. Let’s examine the cross-country flight again for context about this third of the three ATOMs. when he is actually lost. Because you didn’t go with the flow. Instead. You should not need to use the word confidence or power to conclude that the probability of heads is equal to the probability of tails. so that we are not relying on the mean. But we must pay extra care to ensure that what we assume to be true actually holds in real life. Why use a statistic that is? What I’m saying is that we need to expand our scope of knowledge. You should not need statistical significance precisely because of your engineering judgment. the penny would undergo a battery of lab tests. 2) You shouldn’t need a statistically designed experiment to convince yourself that a penny is fair. in this field of expertise. Take extra care to demonstrate that the real life evidence indeed supports our hypothetical assumptions. You fix them. and that is how it should be. 2. Average is conformity. You aren’t Average I am not going to state the law of averages. You have a singular goal. as the only statistic we ever use to summarize data and understand uncertainty. you should not need any flips of the coin to believe that it is fair. the average. You do not average your course deviations or airspeed corrections. engineering analysis.
Similarly. you make a correction. It’s because you have experienced these momentary thermals before. Fundamentally. Everyone else is using the average. How close is close enough? Your judgment informs your decisions. but instead you will accept a momentary excursion to 40 feet higher. navigation. deviations of 3% of your distance are within required performance specifications. You do not need to measure to the nearest onehundredth of a mile to see that you are exactly . As the pilot. standard deviation is the average of the difference of each point from the mean. about where you expected it to be.If you were a thousand feet too high for the first half of your trip (because air traffic control assigned you a higher altitude). You. There are times in tracking tasks where you strive for the smallest deviation possible. Many other times. if exactly on course is the only place you can tell that your navigation is on time. You aren’t. and listening to the radio. Or if you hit a small updraft. The first and most important point is to choose the right statistic. But you must also choose your metric or statistic so variance is insignificant (in other words. deviations are larger and yet still acceptable. clearing. you wouldn’t settle for a thousand feet low for the second half of the trip just to make your average altitude equal the planned altitude. then look at and see that you are passing to the left of the overpass. you don’t make a correction. And you make them. and if it does not achieve that parameter. it is sufficient to see that the overpass is on your right. Sometimes in your navigation tracking tasks. like tracking the centerline of an extremely narrow runway or flying in formation or navigating close to the ground or other obstacles. Except when it’s close enough. Standard Deviation is not Standard Standard deviation is another example of an average statistical parameter. But in the cockpit. you must decide what deviation is acceptable. If it’s close enough. You realize that high gain tracking of your altitude will cause you to lose track of airspeed. you will accept some level of deviation from the selected parameter. if you select a course on the chart that puts you just to the left of a major highway overpass. You especially wouldn’t do it if you were about to fly low-level through the mountains. and deviating one hundred feet left of your route will make it impossible to accurately judge your timing. With traditional navigational aids. have decisions to make about your navigation performance. you do not average your deviations from a parameter. if your heading is 255. the question. For example. For example. You set an attitude that will achieve the desired performance. You are using the statistic that is appropriate for the situation. . “Am I off course” should be easy to answer). you will not fight the attitude of the airplane to maintain your altitude.53 miles left of the overpass. track your progress. the pilot in command. then your metric is worthless. and as a leader. heading. often you will accept 257 without making a corrective input.
It is highly likely that none of these predictions is exactly correct.minute estimate when extrapolated to the scale of tens or hundreds of nautical miles becomes a very noisy interpretation of reality. But in real life. Even the slightest angular disparity is invisible to our eyes. We know that the printed ellipse is not perfectly elliptical.Do not look for what is not there. and the planetary orbits are not ellipses. Being one foot left of the runway centerline does not mean you didn’t land in the center of the runway. it appears as an ellipse. let me give some examples of the use of averages in traditional statistical techniques: . Usually. Statistical significance does not indicate meaningful significance. how much does that truly affect us? And a ceiling that is 300 feet higher than predicted may go unnoticed. cross-country flights with a host of implicit assumptions. We believe the wind will do this. We realize the varying consequences that may result because of uncertainty in predictions. that gravitational anomalies between each planet and others and its own moon and even in the sun’s own gravitational field have created a shape that is jagged and irregular when examined very closely. And it is precisely our understanding that gives us judgment. realizing that the earth’s shape is more like a ball that a child has sat on for too long. But you already knew that the planetary orbits were not perfect ellipses. flatter at the top and bottom and wider at its equator. We must not confuse ourselves with inconsequential statistical conclusions. You do not look for evidence of airspeed by looking out your window for 30 seconds. It’s just the kind of thing that happens when you look at the shape called an ellipse in a magnified bitmap. The second statement may come as a surprise. though. when examined closely enough. Because the number of bits is finite. This is more about variance that masks the evidence. Our experience tells us the fuel flow will be that. The earth is not round. its edges are jagged. We know that planetary orbits are not perfectly elliptical. I want to share one additional example of the importance of understanding the power of assumptions. We routinely set out on navigation missions. Our statistical tools are strong enough to detect the difference. But what does it matter if the fuel flow differs by one percent? And when the wind changes. Your half. It is difficult to truly figure out when the aircraft is directly overhead a given ground reference. none of these differences from our assumptions matter. experience. As we draw this section to a close. and wisdom. Sometimes they do— sometimes we divert to an alternate airfield instead of landing at our destination. Finding a landmark of significant proportion is also very difficult. The weather is likely to change in this or that way based on history and the forecast and so on. You probably accept the first fact. You would have expected that to show up in the evening news. But from a reasonable distance. Any difference from our intended path over the ground and our actual track will add variance to our estimate. we have a general understanding of the uncertainty involved in our assumptions. Our reaction time confounds the truth.
but be intentional in your decision to use it. Sometimes average is the relevant statistical characteristic. Prescription: Use the right statistic. Interpret statistical conclusions appropriately.000 feet. Description: You aren’t average—so why use a statistic that is? Average or another statistic.Linear regression minimizes the average of the residuals. ANOVA examines overall mean and treatment means. . 3. 2. but it does matter when lining up on the runway centerline. The central limit theorem is a statement about sample averages. and planetary orbits are not ellipses. ATOM #3 Summary 1. One hundred yards left of course doesn’t matter on an airway at 30. Example application: The earth isn’t round.
A strip chart is a picture of the data prediction. They could have been replayed in preparation for the test or played alongside the actual execution as a frame of reference. It requires far less cognitive ability. A boundary. In application. A Picture is Worth 10. a maximum dive angle is a solid line on a strip chart. is an easy statistic to monitor. an application of TSM to this flight test technique would result in an analysis performed on a steep dive angle. roll wings level.4 Two wrongs DO make a right My assumption is that no one expects to do a supersonic split-S. abort. a team implemented an algorithm for dive safety planning called time safety margin (TSM). .. and the test engineer. at least digitally.Conclusion Before we close. It is a clearly delineated boundary. his chase. The altitude of the test point probably fell well within those constraints. It’s not the average dive angle that matters. As a result.000 Words Continuing the argument from the last paragraph.. perhaps even 60 degrees nose-low. far less judgment. 4 An additional example with alternative applications of these three ATOMs appears in the appendix. The maximum is a statistic (a nonparametric statistic) and it has many useful characteristics and relationships. They could have been used to support the test and safety planning. It’s not the standard deviation of the dive angle that matters. I want to return to the example used at the outset. The airplane still crashed because of the supersonic split-S. Many hours were spent investigating the cause and developing corrective actions. It is much easier for the brain to process a thought like this “that recovery doesn’t look like the original picture. TSM and the safety planning process would have predicted that dive angle was a critical parameter to monitor. The test team would compute the time required to stop the dive at that attitude. or eject. The strip charts produced from the telemetry feed existed somewhere. It shows the test team what a successfully flown maneuver and recovery look like. In addition. a black and white number above which we do not want a parameter to go.” and then a command is given to recover. the maximum statistic is even more useful. this maneuver had been flown before. TSM and use of the right statistic could have prevented an airplane crash. An F-22 crashed into the ground. It’s much easier to see a parameter rapidly approaching a boundary and call for an abort or recovery before exceeding the limit. TSM would have predicted a safe altitude and recovery. It would have been easier to see something bad developing. The test cards showed these computations and highlighted the recovery technique for the pilot. Therefore. You aren’t Average—Why Use a Statistic that is? TSM would have predicted a maximum dive angle. They wouldn’t do TSM on a supersonic split-S. Let me illustrate an alternate ending to that terrible tragedy using the principles we have just introduced. and pull back to level flight. It’s not the confidence interval estimating average dive angel that matters. It is less uncertain.
” followed by an abort or ejection command. You need to learn to use the ones you have. and validated by the observations. senior management. You need to sharpen the saw you already have. and reports know the kinds of questions that their supervisors. .The more difficult thought process. “Much of learning how to manage workflow in a black belt way is about laying out gear and practicing the moves so that the requisite thinking happens more automatically and it’s a lot easier to get engaged in the game. If the complex processes and algorithms that an organization has adopted don’t fit a given problem. Those creating the slides. presentations. simple moves that are easy to practice and remember. the people doing them suffer from confusion and lack of clear direction. Bad graphs cost us billions of dollars. faster. and cheaper. these ATOMs. They understand the objectives these graphs support and the principles by which leaders will evaluate them. Nor does having paintbrushes make us artists. David. You don’t need another tool. Just knowing how to do all of those things does not produce results. Pictures are incredibly powerful.. New York: Penguin. It’s smarter. It’s not about what tools you use. and right now his airspeed is really high and his altitude is getting lower really fast. Instead of complexity. and this one shows airspeed. like the one your auto mechanic uses. these fundamentals. It provides a common language. 2003. But it is the first step. Getting Things Done.”5 You now have a knowledge of ATOMs. and the new engineer can make sense of them. are applicable at any and all levels: The Secretary of Defense doesn’t need a PhD in math to use them. It provides ownership—people know what it looks like to win and see their role in this process. You can see how prior beliefs (assumptions) play into the interpretation of the data in the first case.. Woven into every fiber is something predicted before the test began. Back to the Question: “What is Our Plan?” Merely having the ability doesn’t make us capable. with no strip chart to serve as a comparison. executed and observed during the test. It’s not about a tool chest that’s big and unwieldy and has a thousand parts. It’s something we can do intuitively and expertly as pilots and engineers and scientists. 5 Allen. too. is as follows: “This strip chart parameter shows altitude. and company leaders will ask. The Predict-Test-Validate construct—or alternatively the chart the course steps of plan-executevalidate—are an integral part of the fabric in the scenario above. We make the wrong decisions by reading them wrong. Having a more flexible and fundamental set of tools provides clarity. It is part of a behavior that is practically hardwired in our brains. We waste time making them. Noted leadership expert David Allen says this. Having tools doesn’t make us architects.
we might have a community of test pilots and engineers and data scientists and program managers that understand these tools. The task is to create an environment that sustains this relationship. question and apply. I want you to build a nuclear reactor. because it's not. You don’t need another engineer or even a statistician. to lead. Nuclear fusion isn’t easy: It takes an enormous amount of thermal energy to build that environment. Leadership. Sometimes another person’s perspective reveals exactly what we could not see. In two generations. You need to inspire them. attempting to create a collision and transform them into energy. I said that I wanted you to build a dam to harness the flood of data—I actually have a bigger goal. Lead the charge. Question the way you make one graph on a power point slide this week. The success that awaits will measure far greater than Chamberlain’s or Abrams’.It’s about empowering your people. As a leader. Certainly atoms combine to form molecules. I want you to smash atoms and reap that almost incomprehensible power. to make one little change. It is far more powerful than the bayonet or that tank. our ability to inspire and encourage and energize team members to take one more little step. We have the time to do one small thing here and one little thing there. we should start with accelerating a single stream of particles. to understand one elementary analytical tool… That is our weapon. to equip team members with these ATOMs. to see your vision and share it with their teams. you are accelerating ATOMs towards each other. but their true power comes from converting them from matter into energy. That requires far less energy. purpose. I want you to combine ATOMs together to create more complex tools and processes. our vision of the destination. You need the people you have to take ownership and responsibility. and vision. to harness the power of fusion. to put the right people together in close proximity and infuse the situation with the heat of motivation. I hope for widespread collaboration. I want the community to discuss and share. Inspire just one more person this month. but we won’t reach that point tomorrow. I want you to understand them at their most fundamental level. There's an equation for it: E=mc2. People forget what the destination is unless we remind them. For now. Take ten extra minutes in that technical review. Give the command. We are waiting: Show us how to transform uncertainty into action. There’s far more potential in a nuclear fusion process than in hydroelectric power. Big steps are not the solution—it is the cumulative effect of a million little steps. I say almost incomprehensible. That's my point. It seems like something we can accomplish. . The knowledge they represent when applied results in an amazing transformation of uncertainty into decisive action.
This paper is copyright. in whole or in part. or the social network of your choice. There is one caveat: You can’t share it with anyone—you can only share it with leaders. not ship steerers. but you have my explicit permission to share it. not pencil-pushers. Feel free to copy it. . mark@multiplyleadership. by email. Contact me by phone. print it. great! Please tell me how. (615) 796-6359. tweet it. If it ticked you off— good—if it hurt your feelings—even better. If you decided to implement even one little-bity change in the way you interpret uncertainty and risk.com. email it. I crave your feedback.Afterword This colorful white paper is a work in progress.
Appendix – Supplemental Materials .
the world had an eerie lime glow through our night vision goggles. It was choreographed down to the last detail. Black as it was. who would say it. an AC-130 watched a black and white television episode for which there is no sequel and sliced hot tracers through the night in preparation for our arrival. confirming our track and the position of the lead aircraft. and hacked the clock as soon as we were directly overhead. crosschecking the selected heading and altitude. I crosschecked my stopwatch and positioned my finger on the reset button. I listened. while my eyes scanned the terrain for a distinguishing feature annotated on my chart. at least subconsciously. what would be said. Somewhere. I cleared in the direction of turn. except for the chime of the clock counting down to our time on target. and what I would do. At precisely 10 miles prior to the next turn. I identified the distinctive landmark visually. altitude. As soon as my eyes found the switch. each member of the crew would check in methodically. And our time deviation never got that large. managing the arrival time at the next point. unleashing its deadly cargo on the world below. absorbing everything. None of them would come within a mile of the aircraft. When it did. The only airplane you could see was your flight lead. selecting switches. if you had one. things you could never predict perfectly. We had to trust that the other aircraft were exactly in the position they were supposed to be. updating it based on a variety of environmental factors. Occasionally the radio came to life with a scratchy voice. Less than one minute to the turn point.A Low-Level Journey – Another Chart the Course Example The drop zone was exactly 10 minutes away from our holding point. The jump seat pilot would command a speed change. It was silent on the flight deck. I knew exactly what was going to happen next. We knew the location of every tower and obstacle. I would brief the heading. With just a one-minute notice. orbiting nearby. I went head-down in the cockpit. An aircraft in front of us was over the drop zone. And they trusted us. Our plan was that precise. and length of the next leg. ground. towards the three o’clock position. Then I would return to my scan: Clock. . We would be at every single waypoint on our route within 30 seconds of our planned time. we could push out of holding and join a line of aircraft. I would need to know when we were directly overhead the point. configuring the aircraft. As soon as we started the turn. Our C-17 was farther back. We could see the terrain on either side and in front of us. a formation more than 10 miles long. led by the MC-130s. familiarizing myself with the surrounding landscape. map. my hand would go there while I looked back outside. the jump pilot commanded the slow-down airspeed and the aircraft commander directed the slowdown checklist. confirming settings. We did. Then I scanned to the left and right. It was pitch black. I looked over my shoulder.
Call switch: “Five. “AD” in the heads-up display. then the sound of a bong.” then release. which for now succeed the higher priority cockpit communications.” from the back. “Drogue okay.” to abort the release should the need arise.” then release. Twenty seconds to go: “Stand-by ready light. airspeed. “Load clear.” sounds from the loadmaster in the back. poised to silence the din of unexpected secure-comm chatter. Twenty-five seconds to go—I see the drop zone exactly as I expect it in front of me. Fifteen seconds to go—I press the ready light and click the call rocker switch.” The aircraft heaves as the load translates. My right hand finds the “call” switch.” . There’s a fraction of a second between every call here for someone to call a “no drop. hurtling through the darkness to the firefight on the airfield below. Post drop checklist in progress. I repeat the process for “two. To my right another landmark—it confirms we are in position. “Drogue deploy.The air was beating like a drum now. leaving my right hand on the call switch and moving my left hand to the switch guard on the jump light. Six seconds. The absence of any signal from the combat control team on the ground is a good thing. “Red light on. Call switch: “Four. Twelve seconds to go—on altitude.” A pause. I can tell by the boxed letters. and deck angle.” My left hand now lifts the plastic switch-guard while I look back outside for anything but the drop zone markings that have been briefed.” then release. The last platform is gone. Ten seconds—airdrop mode engages. heading. I release switches. the waves of turbulence like percussion on the gaping mouth of the cargo bay.” and “green light. Call switch: “Three.” and “one. We are cleared to drop.
I quickly finish the checklist and resume my dead reckoning navigation.New airspeeds and headings and altitudes being commanded. the examiner tells us to turn our navigation displays back on. The tools are not complicated. a chart. The mission certainly is—but it is merely application of the fundamental tools in the chart the course construct that enabled successful mission execution. with nothing but a stopwatch. . without fail. Few C-17 pilots ever became special operations low-level pilots. and formation flight. and the view out the window. every time. We have to be ready to deliver them to the target. but those who did learned a masterful art of chart-making. The world’s most lethal Special Forces rely on precision engagement. learned how to employ the C-17 with pinpoint accuracy. Our simulated GPS-denied/avionics failure training is complete. mission planning. on time. verbal choreography. As we start the turn five miles past the drop zone. a dialogue between the jump seat and aircraft commander. with or without our navigation systems. the next turn point in sight.
This example comes from the envelope expansion test program for the airdrop of NASA’s very heavyweight jumbo drop test vehicle. The solution was to ignore the average pitch angle change and ignore traditional statistical methods. evaluations. The JDTV was too heavy—it was outside the C-17 airdrop envelope. extraction method. Problem: Estimating the mean change in pitch attitude required a test matrix of more than 450 points. elaborate models.mil/news/story. pilots. There was not enough time.asp?id=123201354 . or resources to perform this kind of build-up. load weight. The test program was safe and effective. This reduced the test matrix to one condition. Because of the limitations of the simulator model and restraints on time. The solution was to examine the maximum change in deck angle at the maximum load weight and worst case extraction method. This was not a normal airdrop. Based on pitch angle. parameters include types of parachutes used for extraction and data for nominal airdrop load weights. a ballistic model of the Ares solid rocket booster. and the need for sufficient replications of each test point. money.Applying the ATOMs – Another Example One more example might be helpful. The simulator only includes a model for normal airdrop operations. Statistical confidence in the maximum statistic assured the test team that the deck angle change at all conditions was safe for the envelope expansion flight test. with as many replicates as desired. and resources. there were more than 450 points in the test matrix. and guesswork. The C17 simulator did not have an accurate model of the system under test—neither the test article nor the parachute system used to extract the test article—so the fidelity of the data and the results would have been suspect. expensive and time-consuming simulations. and training.6 6 See http://www.af. means. text books. to demonstrate the design load limit of the parachute system used in recovery. Models and Simulation: The C-17 Weapons System Trainer is a full-motion level D simulator used for aircrew qualification. normal distributions. and the normal models in the simulator were not rigorous enough for planning purposes. Conducting appropriate build up and practice air drops in the aircraft was equally impossible. this was an insurmountable obstacle. standard deviations. Envelope expansion testing was planned to assess the pitch angle change caused by shift in center of gravity during airdrop. Summary: The C-17 was used to airdrop NASA’s jumbo drop test vehicle (JDTV). The test team needed to practice the flight test and define parameters for initiating contingency actions and emergency procedures.