‘TESTS OF MODEL BEHAVIOR THAT CAN DETECT STRUCTURAL FLAWS DEMONSTRATION WITH SIMULATION EXPERIMENTS Yorga Baas Depart Sys Anas ‘ng: Unies Oxf OHIO S567 USA LINTRODUCTION Vilution of System Dynamics (SD) models involves wo general ps of test Tess of model srucure ‘andes of model Behavior. Yet, ine SD models ae casa modes, the essence of model vali Us slwaysinstrcaral vali: "Right bebavioe othe ight essons. (Te traf SD mode! vay has been cused in various SD iterate. For example, Forester 1961 (Chapter 13}; Frreser 1965; Foreser and Senge 1980; Bell and Senge 180; Richards nd Pugh 1981 (Chapier $ and 6) and Bras 185), ‘Sandard behavior ss, which compare the mode-gneatedbeavionathe bere ference bedava ae ctrl "weak in SD content sine the can no Spurat spurious bhava accuracy right behave for ‘he wrong ress") fom tue behave val. Soc ets provide no srucural information stress, ‘onthe oerhand, are ‘songs since they evaluat the model structure direc. Bathe ir major cawoack shat they ac infomal, quale, hece det communica (See Forse and Senge 1980, and Richardson and Pgh 1981 (chp ) for ome specif eho snd seuctue tes). Tas, seen ike tid ype of test” woul be mast appropiate foe SD valdation purposes: Ouatiave formal ‘etait hat can provide some sacral iafomaton rata eentd” beavis) Tmeresingy, soc ests aad ext inthe SD vation "reper. (For example, "Extreme Condition” slnulatos, “Behavior Sensviy” esting, a Forester an Senge 1980 and Richardson and Pugh 1981), But these tess ae usualy sted slong wil ihe bebavir ests which vasermins the mr dereee thst cists etween the former andthe ate, nts paper, sing simulation experiments, we demons tha he “sracrly-riena” behaviors another tania bevir este dierent in a fundamental sense. We show bw the src ‘oteted behavior st ca help ignoslremave sutra ws, Ths, we sogest fn sche be ‘etd a analyzed by System Dynamics in more dei I hoped tha the ets wil be ipoved, stanrdze and iplementd part ofl he major SD simulation software, 1 EXPERIMENTAL PROCEDURE Inrde to make conto experimentation posible, weft build a model that describes the dys of hyphal epiemicdcase, We west this model 3 "pete esl sytem’ an he ul tee diteent, ‘nodes oft, delibertely incorporating diferent sca asia each on of them. Taro simlaon ‘experimen, we demons that ) models with major structural faws can generate highly acute behavior terns which woul paste "weak behav st, a i) the "sacral rete” behviotetscan ‘dct such spurious bavi scurcy ae ep noove te "béen” tetra. Tre seta. ‘end tehvior ss wil be astae: A- Esme Conon st Assign etme valies toslected rameters ani compar the mode geared behavior he observed (or "aici behavior ofthe ‘stm under ihe same estrone contion B The Babasie Sans ts Dtermine hoe pameters 9 stbich ke model tans dplas igh esky, and ak fhe el sytem woul exhibit sir high serstviy.C- Mii Behave Peition es Is sometimes posible wo fing data aout he behavior of ‘ome "oof version of he al sstem (vi could be dv 1 parameter modifications o struct2 ‘moilexton). The, est if be modes ben generate be sume “mealies behave when simu with sini modifications (Alo he above he ess wee fst suggested by Foret and Seage 1980. Ou ‘efinitons ad emiology given above ace somewhat diferent fom thks, but tse dilerences are merely cooveatona ot eset a ny a). [IL THE SYNTHETIC REAL SYSTEM Asnatoad above, the satire system ved inthe silaton experincats desis th dynamics ‘ofa hypoieial epidemic disease (Figure), The Bow diagram consists of two majo sb setae: Ft, thee te op that "rexyel"the piven populatia, hough he aro tages ofthe epitemic eck: Sesceptible opuatoa > In ced Population >» Contagious Population > Contagious Population Recopazable > Population Having munity > Susceptible Population, which loss the loop Infection ‘use bythe Contains Population contacting ("Cott Rat") the Susceptible Population. Tio (on the eft half igre 1) eases its variables exhibit an oscar behavior pattern. Asse in Figue 2. tbe vera pido thos about 32 months (The asdomness that i apparent in Figure 2s cbse ‘by using to Normal random variables nthe model Infection Pex Contact and Recovery Tine. Tae 0 ‘chav pters depicted by he thi ine aed the thik one re gene by smug the same sytem with two difereat nose seis) The send major sub statue af the mel consis the "population growth ‘oop, dept in the right aff Figure I. Note hat, exept Contagious Population Recopnizale al popstion sub-group ale the Tal Conception Rate Thee population grow ops reat postive wend nal popula sub-groups, nha the overall bebivior othe model ons f epidemic sillations supernposat ona ten (Figure 2). The uit vari poten Fig 2 "Cantpous Population Recognize” (CPR) To kop the experinents laivey spl, we wil always astm hat CPR the only ‘val ouput vaca. (Doe space mia, we kip te ery asin portion ofthe bebo ates, {prove ypc segment rom 010-26), In adton ois reference behav pater, he yale system goers wo mofo secur adetions: When ll grow paaetes are sett0 220 the stem exhibits eclations rund a constant mea, ice al growth loops ae infest remove Te epidemic osilitions ein xcnly untied.) When the mae Pio (P) e135 xtreme lage umber which racicaly meas "permanent immu" hen he egulc pene ‘canons dappear. The modified bbavor conse of “epidemic break ov which our with quit ogsn vvale itera Theft pene stra sbut 10 and ends at =40, the second one starts at 1-130 td cde a ¢-18 he hi one tra 205 and em 12260 (Tis behav is asta ia Figure > ty he soli line Due o page rexritons, we arena able o provide he compute generated raps. We do othve pice to provise the made ogtions ether. The nerd rede refered to Bais 1985 or 1989, {or ore dtd information) Fly, seneviy of the bebavior of the syatete stem tis parame. ‘rts examined. The endamenal patersof eka prove tbe uke laseniveo mest be stem parameters. Changing he liye of he rexyling loop (sch mune Perio ox Recovery Tis) by "ch as 100% doesnot cute the stave of he yen 1 i from one mde eater. The same s tre frthe parameters fhe grows oops (een a erty Fracon). Changing the above parameter aes ony the specie numeral vale of behavior characteris (chs inreased ample or sbi tld), Te only wo paramere o which the system exhib significant remit arehe ones belonging © "tenon inearinfesioa Rate formesion TRKLAPCCRK, whete CRK=CFSPR*CEK oo...) Iacressing the vas of IPC (fection er Coat) or CF (Conte Faction by 50% or moe resus in ge oan that damp oat mmeciael. Decreasing citer ofthese by SO resin unstable growing cillatins. Tas, he oly parameters which he sate system sensve to are IPC and CE IN SIMULATION EXPERIMENTS Tati scton, we ld te dierent made of he yee elsyiem. Each model wilbuve diferent structural erolomisons, hence fering ears of val. We asune at the modeler observes the “el” betavo of he outa varible CPR, forthe period =80 wo 200. (Fr smplty, we assume that his ‘the only variable ed in bebavor testing In anal vabition proces, the tests cussed below woud be applied mare than one varia). “Masel nth model we sme tha the ai eaves oul the population growth ops (le cate righ alo Figue 1). Lacking accurate infortion te meer perbapsaribts the observed growth of (CPR immigration. Tas he observed effec fhe gow ops replace by a single externl inp ‘ction, eximatd tote about 23 people mon Aloe eqution and parame ales ar exact the same 28 the rea aystens (ence he nie sequences ad the observation ec) Th nial eondioas at 0 are ‘ls sesumed ote known tothe dee, When Modi simoltd, CPR exhib the Bebavir pater, represented bythe thin ine in Fig 3, (The thick line corresponds othe ea” bebavior, with 3% ‘ofseraton eon) The agreement ermee the “re” nd Model-geerted pairs is obviously very goo, Conparing Figure 3 wo Figure 2, we see that even the real sstem self does no replicate is own behave: much beer tan Model does wit fete! oie sel! Te behav of mde wil ps alsundard (Cea tess of behavior comparison. (Soch as comparing the periods, amples, ends For detail ‘cls of hese, se Bale 1985) Nex, we jet Mol 1 the three seucurally rented behave ess: ‘A Exum Conon As an exee con, we set RTAVG (average recover ine) 108 vey lange number (ke 10000), which reality mens no recovery. The sulting behavior of Mode i ‘eprsete bythe died linen Figure) This consis fa sharp increase followed by mk long.ecles ‘operinposedon end (The chavo pater ilistated ia Figures 4 hough ae ns pecise eee ‘hey were hand-drawn in order conserve pce Ye, they Serve thei purpose by conveying te fdementa panne often), The tehavio of the syteti reals ander fe same extreme condtn, i asa itfeen (ine in Figure a: kina sharp nctas, followed by a seay dectine ase by eth) a eat, node to be able apply suc est he analyst would ave to ancpte (sing expert opinion theory and dia iestblshed liertize et) he behav ofthe eal system under te exeion, One te est sap, and he mode is ne aay mst we reasoning oa e «ae offal: When RTAVG is mde exe mel re i the syntete realty, CPR cots ery lrg ‘roporton ofthe overall population. Tas, era resa in retical lower ih rate, bra CPR it overt group ith sytem. Tat why CPR fist soos up and then ects thoughout the simul. fo Made, however, the et bi ates independent of CP, since te former formulated Simply san extemal inp. 'B- Beavir Ses at The sentivity of Mode parameter i een the same that of the symteticrel sytem. The model passes thi est (C- Madi Behavne Prison Ts: As mentioned shove the sate system gre the “ectest pide" typeof bebavor (Sli nin Figure 4) when tbe munity ide permanent by sting IP (immune Pei) oa very large ume. Assume that his “modified” behave ofthe sytem was 0a 1 ss. Aly the sume modfcao o Model. which exhibits the behavior represented bythe debe na Figure 4. Te only diference between he two grap ith, athe rea” behav the ine interval between _piemicbreakous becomes shorter and shor, wherein the mode remains const nthe Model 9 200 cine) igure 3, The Behavior Pattern Generated by MoscI“l (This Lise), and the one Glgerated by the fyeenerie teal Syeees (bier Lin) Se He ilo Aaa SSH a aS_peveraed beta, (Because the shoring ofthe trl cused bythe ever-ieesing population gowah ‘te, which merely a constant in Moel1)- Whether this sable eitlerence inthe mie behavior redision wold ost doubt on he wees of Model! would depend on is intended purpose. MadeL.2. nhs experiment, we incorporate amuch mare sous dling eo th the previous ne: “Tae mun acqied fer having the cae sumed to be permanent Tas, the crcl rcyling™ loop eft al of Figure 1 bck ie Modl-2. Te analy, who srs ot with hisincorec assumption ‘of permanent nmin, ate observer hath rel envio silstory lst bad modeing practice, ts assume thal th analy then ties" tune the remaining src of Model, noréer 10 [eters caclaton out of One way of schieving hi make the population growth mi high eoough ‘wich would reste eaogh flow in the Susceptible Poulin, ii osciatins iia tthe ‘hserved ont, Naturally, most fhe pameters nd inal condons ofthis del woud have 1 be "ad ac" fne-aned inorder obtain the righ xclations (ee Bus 1985, for specif deal), At ibe “Mode-2 manages to geerte a CPR teaver very smart the ne exited bythe "observe" CPRin gure 2 (Duet space estictions, we rena prsating hi parcular rapt). Once api, afer 009 “fg troy, be eto of Model 2 pases the “weak” tests behav comparn, inspite ofthe fact shai very major striae error (ee Bari 1985). Wenext apply the tee statraly- oriented ‘hire to uneoer he strc lendxutey of Model-2: ‘A Extn Conon Te We repeat te exreme conto st dne fr Mode: Set RTAVG to very large numer (90 recovery) We aledy now tt under his extreme codon the sya system _rernes CPR bevir tht conse sar inert followed bya uniform deine (Soi ine Figure 53). Whe the sane exten codon aplied o Model 2, CPR generates behavior pater ery sia ‘he bane rn: Orion (ith a periad of abot 35 roperimpoed ona poste trend (Figure Sa, shod ine). Tati he unmesal patern of ehavir of Mode-2 change very ile under his extreme condon. ‘Te reatonsobvou: Since Model leaysuumes permanent immunity, making RTAVG extremely ge oes not break any op! The population growth oop which ae esponile fotheesciltory end tbebrviorof Mosel aril ace (The powth of CPR becomes naturally much prince here is 0 recovery under he extern condion). Modeling thi extreme condi ist woul be taken byte ‘modelers nation of stor eo. Invetgnting he cause of fare woul ure sages the lection of te stata et. 'B- Beavir Senki Text Asan example of sensi tetng, we decrease he val of FF (Feiity ration) by SO. As esa oh csage the only aonb change i the behavior ofthe al” CPR one flere en spe (Fig 5, oer). Tat the behavior pater ofthe sytem 8 insersve otis pata parameter. Buin Model-2, when he see parameter change indice, we se igure 5. tases of: (e)Extcene Condition Testing wich ROAHo=1000000, (b) Senasesysey, ‘Rielng wtih e 308 becreasing of PF, (eModitied-Beharior vith "zero growth" CORTESE? sjeten Sehardors bashed Line! Rodel-generated Bebavi08.‘hate behavior of CPR changes drastically: is period increas rom 3560 and ios moe Ikea "recringepdemis typeof tehavor (se Figure Sd nes). Tis extreme and inpropr sensi of Mode: 1o FF would soggest tha the model mayb licking a major soir, thatthe ane beter ‘was obtained by exesve “a ning” More pci, he tt gaz ht he excel fe-tned subsircture(glowth loos) inl the parameter FE, (C- Mati. Behavior Prediction TA motifed-behavior ofthe synthetic sytem mentioned eae it ‘cits around a constan mean (Sl ne in Figure Se) The sre exis this behavior when here it ‘0 population growth (ete Fert Fraction F ad all det aco 0). The Moified-Bebavior Prodi tet woul! a if Mode abe o generat the sme mie behave, Wha the "2- growth" ‘mations re one in Mode!-2, teresting bebavor of CPR ie single growth-then-ectne (fae ne in Figure 5c). The obvious ale of Mode-2 comes rom the fat thal he population ops ia this model are "fine-tune to peneate the silts, and when thee oop re removed he scllaions completly disappear. This Modiied-Behavioe Prediction xt woul! suggest the aaa that Mode: acs the coset substrctureta responsible forthe slatos in the bei ofthe rel syste. ‘Model. Finny, na thd model ofthe syne system, instead of incorporating single obvious ‘structural er, we incorporate many sal "ypc" modeling erosions. Such eors woud lacude electing some mor eps pacing the ticipate effet by fnctonal (TABLE) latch, eving ‘ut some ofthe accumulations o lays representing ome high-order delay with fore ay et The folowing structural erorvomisions wee incorporated in Model) Two the-nder dels, Population Infect an Contagious Population ne oat sd ther effet ae aprons by an increased Recovery ‘Time and tnmune Pid i) The two secetive deny, mune Infants nd Conefved Bi re replaced basing ong, he length of whic equal the sum ofthe two origina delays) Al he mao ops ‘eprsetng he "death tes" are removed. Thr overall effect extimated an represented by a decressed Sopaerage nt" growth te) The re! nop-linea neton Rate cee (1 above) shang. Sach an sherentve formulton wo nde to compensate forthe effet ofthe to missing delay, Ppuation Info and Contagious Population Te Infection Rate a Models given by TRKLAREK*SPK, whee new variable Infection Rate Fraction (IRF) depends on Contagious Population Frcion(CPF: TIREK-TABHL{FTAB,CPFK.000 16 012)+NO FTAB=0.0.0200.0400.0659.09001101.1251.140)0150 (CPEKeCPK/(CPK PHLK+SPK) ‘The compete ow Diagram of Models given i Fig 6 Noe ht hs mode has coosiersbe es ell and ever varies ha bot the steve sytem, andthe wo previous models oft I Mode, we sume thatthe mai output variable C® since CP and CPR are combine into ingle variable. The lngotan feature of Model isha abough tq small a simple, ides conti approximate) the ‘wo majorsobstracter of he Sytete Systm esponsbeforth oii ad wed putes. wil ‘therefore presen sb ality guestons tas he previous we model. When Model simu under ‘normal coations, is behavior pater ronce agin vitally ndtingishble om the behav of be ‘yet tem (or complet lat of eatin nd behavir patra, ee Brit 1985) Ths, he ase-u “behavior of Model easly pases the stander (‘weak’) eho compris sts (Bars 1985). Net we ply he tre suctrally- ened bavi et: ‘A Exum Coins Te: A nese conto et, when RTAVG i et 0 100000 in Model, (CPexibis prs growth pater (dashed lin in Figure 7), We Know Srom previous experiment tt, under the sume exeme codon, he yeti rea system exis gowdh-iben-decine pater oi ie in Figure 7a). Te src reason behind hi are of Model ithe fat hat dest sings teween CP and CPR (Contagious Population Resgnznle, which non-eproducive athe sya ‘stem, I Mode-, both the Infection Rate ad the Birth Rate depend pinay on CP and SP and sne an ‘eureme increase RTAVG rie the average eve of CR, the euling bear a persistent growth. This ‘eteme condones warns us tha, depending onthe niended ws ofthe model the CPICEReisinsion may provetote acral on. ‘eave Sesiviy TAs aca est, we doable the ale of RTAVG, Compare the sll ant «asd curve of Figure 7 The resting behavior of he synths sytem oe of stronger eciations with ap nceased hope (tout 1.2) Mal on the ther hind, exhib very mil ceilations sperinposedon& ‘ery strong slop (ebout 20), Model-4 play "wrong esti 8 sul of doubling RTAVG:Isead of exhibiting stronger ositons the ater sos dsapea.This i again cased by the fc tat 3 CP;CPR pct 7 The Yoet-senerated(Oesed line) and Sytangnesaed theviorFuttarne (old me Line): (a) Under the Extreme Condition of RTAVG=1000000, (b) Sensitivities cotinction is mate in Mode-4 Since Ineton Rate dirty depends on CP, yon doubling AVG, the ‘noe lop tat generates he Ineton Rate mints the es spl) lop which the main acbaism ‘hates ie oelations A anther sensi test, we increase the Immute eo (IP) by 50%, The synthetic sytem splays very sensi to this, Thererting behaviors exentillyhesame asthe taser, wth sight stronger oslo, sn lig longer pcos. Model-4, on the ee and exibis ‘oo much sensitiv, teresting behavior consisting of very large and growing oscillations. Thigh and improper esti oP indicts that he ndrof Model 0 fw antlorsome impctaatoops re ming: Index Made-t aso few scumulatons nd ay rod the eying op (C- Maid Behavior radio is Mae able generat the the constan-nean ocr mode, by sting FF=00. Wi some minor probit ale ble generate the bass shape of be "reerent” epidemie made of behave, ty stig IP 1a ¥ry large aur. ‘Let silly emphasize tha, we do nt suggest hat models be simpy “eet” "scepte ypon ‘plying te above (x ober) tess. T try oid a "esr 0" answer to adel valiy ot aconsucve th ase dels should he viewed "eto ent vad wih expect given problem and Purpose. nt. neo the songest rgument forthe srt aint Behavior ests would be tha they ‘am elp he asl cover and remove te stctral faa hid behav inadequacy, andy inrene Ihe vali of given mode. \V-CONCLUSIONS ‘Te above experimental rents demonstrate thee are wo fundamental dierent ypes of behavior ‘validates "Weak" bebavor ss compare the behavior ofthe model oa gveareeeace behavior ofthe rea system, an ean nt separate behvirvalety from spurious behviracursey. Such ets provide ostactual infomation, Sacral. crinted "ron behavior tes, onthe oe and can provide soe stra informatio, pence belp uncover te structural ws of given model Expeinens ate tee such ests: Extreme Condos tt, Behavior Sensi tet an Modified Behavior Predicion tet Result show that these (doe sla srctral-orened behavior es canbe very weal nemunig the éegrecof valid of given model We suggest that such ess be atepoied ad spices be analyze by System Dynamics inde Is boped that sacuraly-orene ests wl be improved, oemalized ant ‘venuly implemented pt ofalthe major SD simulation oftware REFERENCES ss, Yaa. 1985, at, Yan. 198 “Mai Ts Vk of Sytem Dy The of Sinton Mae’ [arog oumal of Operas Research Fortconing” eles at Senge, Pr. M980. ‘Mehos for Enhancing Refwabiin Sytem Dyaunis adsing, io Forse iy, Weta coe, Systm Bynum, Norlin, Fomeste, ny, W- 1961 India Dynamics, MIT Press, Cambridge, MA, Faster, W196. "A Response to Ansolf and levi", Management Sees, Wo. 4 pp. 61 RSE op De Semy ey Mg ns oe Bag Cafe Sten Dyas Mode in ister. Joy Wc, Sem Dymus Nessa, New Sore secrete Cantedas, MAY