‘1116123, 202 PM Bayesian meta-analysis in bemsll|A, Solomon Kure About Blog Pubs Books Workshops Courses Support CV. Bayesian meta-analysis in brms-IT BByA. Solomon Kurz October 16,2020 Version 1.1.0 [ited on December 12,2022, to wse the mew a5_drans_6¢() workflow, Preamble Ii Sexton 14.3 of my (2020) eanslation ofthe fist ction of MeEleath’s (2013) Statistical eshinking, included a bonus section covering Bayesian meta-analysis, or my (2020) transition ofthe second eition ofthe ext ( Malsuls 2020), like to include another section on the topic, but from a diferent perspective. Tae fist time around, we frused on standardized mean difeeness. This ine, I'd ike o tackle odds fatios and, while we've at give aie Bi oF a plug for open science practices “The purpose ofthis post is to gresent a rough dal ofthe section intent ack this ‘which covers measurement ero. Ifyou ave any constrictive rticisms, please pas them along ether. fom onto the end of Chaper 15 (Missing Data and Other Opportunities), the Gi issues frie ak oron Tau ‘Here's the rough dat ‘Summary Bonus: Bayesian meta-analysis with odds ratios Hil # spened and made before this section sirary(sloyeense 1ibeary teens) iprary(egdart) brarytedetatsy Library (oreo) Hbraryttioyenyes) omen EEE “panel-grid = elenent_plank()) , 4 to reset the defutt goplot? thone to Lt defeult parameters, 1 execute “gupta! thowe set(thene_groy())" and "gpdarks tere goon defoutts() 1f your mind isn't filly blown by those measurement-rror and missng-dta model, let's kep building. As it tums oat, meta-analyses are offen jus special kinds of smulevel measuremencartor mide. Thus, you ean ise bene: sore) to ft Basin meta-analyses, 00 [Before we proceed I should acknowledge that his section is heavily influenced by Mati Vous great blog post, Metaanahsis isa special case of Bayesian ing. Since neither eitons of MeElreath’s text dvellyaddess meta-analyses, we'l lso have to borrow 4 i from Gelian, Cari, Ste, Dnson, ‘elvan, and Rubin's (2013) Bayesian data anasis, Tid edition How do meta-analyses it into the picture? ill ‘Let Gelman and colleagues introduce the topic: Discussions of meta-analysis are sometimes imprecise about the estimandsof ttre in the eoaysis, expecially when the primary foes ison testing the rll iypothess ofa effet in ay ofthe stds tobe combined. Ou focus i. estimating mearing Tl parameters, and fr this objective there appear 10 be trae possibilities, accepting the overarching assumpton tha the studies are comparable some broad sen, The Rit possiblity i tht we view the ‘Mudies es identical replications ofeach ote, ithe sense we regatd the individuals all te studies independent samples fom «common population, ‘withthe same oucome measures and so on. A second possibility isha the studies ae so diferent thatthe esl of any one study provide no ‘nformation about the resus of any ofthe others. Athi, more general, possibility is that we raed the studies a exchangeable Bu ot necessarily citer identical or completely unrelated in other words me allow ctfrences from study to std. But uch tht the difervee arent expected a prior! to have prediciable effect favorig one sty over another . thi third possibility represents a continu between the two extremes, a ts hi cexchangeable model (with unknown hyperparameters characterizing the populason distribution) tat forms the basis of our Bayesian analysis. ‘Tae fist potential estimand of a meta-analysis, oa hierarchically structured problem in general, isthe mean of the distribution of effet sizes, since this represents the overall average” fect aos all sues that could be regarded as exchangeable with the observed studies. Other posible esimands ae the effect size in any ofthe observed studies andthe effect size in another, comparable (exchangeable) unobserved study (pp. 125-126, emphasis inthe cxiginal) ‘The base version of a Bayesian meta-analysis follows the form. -ntpfsolomonkurz netify.applog/2020-10-16-bayesian-meta-analysis-mbrms-id ano‘ens, 202 Pm Bayesian meta-analysis in brmsil | A. Solomon Kure yy ~ Normal(@s,0;), ‘where yj =the point estimate forthe effect size ofa singe study, j, which s presumed to have Been a draw fom a Normal distribution centered on 8. The date in ‘meta-analyses are ypeslysatisticalsurmsres rom individual te. The one cleat leon fom this chapters tha hose estimates themselves come with error and those eos shouldbe fully expressed inthe meta-analytc model. The standard eror fom study jis specified, which s also a stand-in forthe standard 65, 68, 102, 126, 145, 263, 267, 298, 360, 308, 350, 356, 276, 432, 438, — on S vorcatte ehes Mest! ea®, SYeaty "vests Svebey shots “yest, "ves, ey "ves", Nes oa § factor chro "Sieeeerect™, “Steet ffects, “sikstnct®, “Siaeterecté “siaatsfect™, "3 wesc ches SW slated Grass" Portuguese execution Allapal_reenv?, M2 = HS Source ehes “beediias; “beast Tier, sorasilia, swtlftedlour®y Selb fnedaate, "Bey oe S hourtsa EBRD apg bey Oy By Ay ty De de yy te Ay dy By By My 2 2y Oy Dy Te SES hosst tea ab1> 38.a5e,"s5 08, Serdss, 24625635, Soréa8, 58.637, fata, 2 ot § haueach EabD Wt, Ay NA, A, Ma, ‘My RA, MA, Mt, A Ay Ny S$ houeacie a Sab13 Mey we, MA) Nay A Me, WAY es A A ‘s4 § Source-Prinary ‘chro “brasilia", “beasilia", fm § Scarce Seconcay cahry “beastiio"y “beast lia", HE Country fs "beers “aractl", "brani" “Canada”, "canaea", "Coady "Canada", "ca oes Coenen fehes “Social dna hark Payenology Geportnent, Univorasty of bravia, oF, Braet HS Longuete ehes “Portapuese”y “Poreupuese"y “Porcupuese”y “English; "English", “thelish~ oe S betes coos aa ath BES Execation ahs FES Stbjectneot Lehr iateeads, “ne fa § Sctting Eohe> “ln 4 dsdroans, “In a claseroce’y win a claversor Sa $ Penett SERED SSP the ahdae study uac‘an the snputer(oneeat nye: conser /d 1 § Stooyordenw —“chry “hacser oss. siatel| ottenseetch/orohn[kayliraur fencers aL ongel ane St § sosvlorcesn —Schry sibs elobar gelf-esteer STot Ho" Mooe{ TD" subgecttve welibeingl 9" Doepue Ss Ronoer bana Gibas tt oid garded htt det MES aralystsstyoe chs “onder") “Orders Marden", "arden"; "Order'y Mérdore) “Sedarss “Boden! = oe § Sibeet fetes Salty aly Salary saint, Salty, vailey canny sma, alley “ait fat Hi $ Gesecincge gh Tue,"TRie, TUE, Thue, TR, TUE, TAUE, TRUE, Te, TE, Thue, TR, T> my SH te Es Les 8, Uy dy Bo Ly Mey ety By 8, Le dy Ody Ay By Ly de MS toe Ree URES PERLE Lge ee ees ‘The total sample sie is N= 6, $42. nw {istince (un) 29% count) wee a ciobles ta Allcate are tobe included -ntpfsolomonkurz netify.applog/2020-10-16-bayesian-meta-analysis-mbrms-id 20‘1116123, 202 PM Bayesian meta-analysis in bemsll|A, Solomon Kure count {case-include) “The data were collected in 59 locations with sample sizes ranging from 34 10325 ne Feine(Lacstion) 29% Srrangetarscto)) St 1 Unversity 0f toronto, Scarborouen 4 5 Cotvos Lorand University, in Budapest, Hungary St § Soparanent of Social Peychology, tlburg Utsverstty, P.O. Box 99353, Ti2ban 7 boparanent of Psychotogy, San Diego state university, Sah lego, CA/S2i82 1 beparznent of Psychology, Pennsylvania state University Abington, AblPgtOn, PA 19002 Our effet size willbe an odds ratio. ‘Here's how Klin and colleagues summarized thee mary results: Inthe sxgregate replication sample (N’ — 6, 842 fer removing pricipnts who responded i ess thn 38,7148 of patpats judged the action inthe foreseenside-ffect scenario as permissible, but only 17% of paicipants inthe greater-od senaio judged it as permissible Te difeence (1.28, 1.41] The replication resus were betwoon the percentages was significant, p = 2.2¢~*8, OR = 11.54, d ‘iin d ~ 2.50). p45 ‘Hew is the breakdown ofthe outcome and primary experimental condition, which will confiam the two empiticl percentages mentioned, above Eount(variable, factor) rovpaby(tactet) BX sorstatgorcene * Teo # 8 / um(ny) see a ctoble: 44 fa N Groupee factor (2 fe Vevtabie factor fn percent state Slieetteces tase le “Though the authors presented their overall effet size with a p-value, an odds-ratio (OR), and a Coben's de, a kind of standardized mean diferenes), we will Focus oa the OR. The primary data are binomial counts, which are wellbandled with logistic rogression, When you perform a logs regression where a control condition is compaced with some experimettalconiton, the dllerence beweea tase conitons may be exprtsed ss an OR. To gel #yense of wat ati, e'l Tistprectice iting a logistic rogression model wth the frequents ging) Tunetion. Here are the ress based oa the subset of data om the fst location, fino « gincy = factor, family = winoniat(Logi), data = h AE Finter(loe =) sunmary(glne) It lnormune = y ~ ractr, fanly « binontal(2oete, date = ha we else oS fi asut ect 1a er a.8hte ” Estinate Ste. teror 2 value PG lel) St cancarcesty —“SaeStoe orSeas “tlabe Zoneecan ove oitwrsionpraater for bnoialfaiy taken 9 be 3) wots cevtances 139.47 on 182 grees of frosdon 1 nunber of Fisher Scoring Sterations: 4 Sus ike with brms, he base-R gia) function ret the results of logistic regression mode! inthe log-odds metic. The intercept isthe logos probability of {electing yes inthe sy Tor partiipants in the Greatencood condition, The “actorSideB fee’ parameter i the dllerence in logo probability fr patents a the sidetffect condition Here's what happens when you exponentiat that coefficient, coat(elne)[2] 0 690) -nips:fsolomonkure.netiy.appbog/2020-10-16-bayesian-meta-analysie--rms-id 35, 95% CL consistent with the dovble-effest hypothesis, nd he effect was about hall the magitide ofthe orginal ( 30‘1116123, 202 PM Bayesian meta-analysis in brmsll|A. Solomon Kurz That, my fiends, san als rti (OR). Odds ratio ae simply exponentiated logistic regression coefficients, The implication of this particular OR i tha thse in ‘he sideteFece condition have about 10 times the odds of selecting Jes compared o thos inthe Greaterood condition. Inthe case of his subst ofthe dat, that's 19% yeses versus 69%, which seems ike a ange diference tome Meer(loc = 1) 8 Seint(eaabiey Vactor) #98 raupsby (factor) OX oxstatvarcent * Tel © 9 sum(ay) 398 Palter(voriaie == "Ves" BEF Groupe: factor 121 Log-odds, odds ratios, and modeling effect sizes. Hi Thoughts common or rs ‘aceon way we mod! hom! dta Wits the logit ink The logit nk transfoms bones [0,1 parameter space nto an unbound paruetr spice ranging from gave to postive ify Far us Bayesian, tao provides cotextin which our parame ate approximately Gaussian. However, exponent those approximately Gaui loads coeticients, the resting ds ratio aren't so Gaui ny more. This 8 why, evn four timate gol so xpress a mesa-aalyc effect as an OR, we Wat o Work Wilh effec sizes in the log-odds metic. It allows us 0 Use the Bayesian meta-analyicfamewotk ouined by Gelman el above, chers to expres their effect ies as orto, we don't want to work diet with odds ratios na meta-analysis. Why? Well think hopin align) y_j& si aperatorname (Normal (thet j, sigma jhe j sim operatorame| Normal (ens ta), endfalign®) ‘where isthe point estimate in the jth study stl inthe Jog-odds scale. ARer iting the model, we can then exponenite the met nate parameter pinto the OR Compute the study-spcific effect sizes. Hi ‘Our data fom the Klein et al replication sty includes the un-agaegated data from al ofthe study locations combined. Before we compute our mets-analysis, ‘well ned o compute the sdy-mpecii effect sizes and standard errors, Here we do 0 within» nested ible feleee(loe, y, (actor) 19% apsetelne, e3ta = .))) 39% 159) Selece(seta, “gin) 38 Sener (coe) 5H ‘itertiere tate tf(is double, rund, digits = 2) Inthe estimate columa we ave all the yj values and std.errar contains the coresponding values. Here they ae in apo color & variate pat(ortion = °°°)09151 ieploe(aes(x = eanspeint (color ISbe(e = expression(stgra[itac() ("10-0086"), Y & expresstontatalieQiJi}eC lop-ose5"))) hitpsisolomonkurz nett. apptblog/2020-10-16-bayestan-meta-analysis-r-brms-i!sniv829,2.02 PM ‘ayesan meta-analysis in msl [A Solomon Kure alysis, [Now ae dats are ready, we cat express ou ist Bayesian mets-analyss with the formal Fit the Bayesian meta ein aign®)exestinate)_j sim operatomame{ Normal theta jy text{tderot)_) thet j& si aperatorname (Normal (a, au) & sin ‘peratomameNormal}(0, 15) una dim \operatorame Exponenil)1), end falign*) ‘where the last lines spel ut ou priors. As we leaned in Sexton LLL the Normal(0, 1.5) prior in the log-odds ypace i just about flat onthe probebility space Ifyou wanted tobe more conservative, consider something ike Normmal(0, 1). Hete's how to ithe mode with brms. ‘entaata + elms, sStinnte Pse(steverror) = 1+ G1 loos efor = efpetortnarnai(o, 25), east = Eterceps), (rlor(eapenential(:), class = 32), xen = 2908) aarmp™ 1080, cares 2 chan eed 2 15) se) is one ofthe brma helper functions designed to provide additonal information about the citeion variable. Here informs brag) tht etch estinate value has {nassocated measurement eror defined in the sts.er-on colar, Unlike the a) function, which we used earlier the chaper o accommodate measurement cor nd the Bayesian imputation of missing data the re()hucton i specially designed handle meta-analyses. ef) covtains a sspra argument Which i eo FALSE DY “ene ng Oversisy of Selene hitpsisolomonkurz nett. apptblog/2020-10-16-bayestan-meta-analysis-r-brms-i! eno‘1116123, 202 PM Bayesian meta-analysis in bemsll|A, Solomon Kure teeation teestion teestion teestion teeation teestion yee Depertnent of Experimental and Applied Psyetology, Wo Atsterdan, A06i8T, Ansterdan, The Netherlands” = “WA Anstercan Departnent of Sociai Psyehology, Tiiburg diversity, 9.0. Box 9653, Thibung, S066 LE, NetTerlangs" ~ “Department of Socal Psycholog University of Lasarac, Suiteeriand” = "Unversity of Lovsanne", lnsversste Je poeiers, France” = “Universe de Pestiers"y “Turk India Workers” ~ "rTurk India Workers", . “Tiniversiy of tinchcetery Minchesters ranpehire,, England ‘Dashisho Unversity, Kyoto, Japon? "Doshishe University", ‘tletonda Unvers:cy of wellington, tow Zealand” = "vieton.s University of wellington", Universidade do Porto, Portugal” = "Universidade do Porto", “lnsverssy ot Belgrade, selgrase, seroia® = *unaversSsy of Belgrade", ‘Santiago, chile" = "Santiago, chile’ . ‘ationa: Astonoous University ev Nexica Jn Moxico ity® = “NatiotalAstoroneys University of Mexico", S= ‘Acavenla sinica, Talian National Tainan Marval University, Talwan* ~ “Tetwan National Taluan Normal University Hig avert, star Terkey” sal beri "arerican University of Sharjah, United Arab Evirates* = "Anerican University of Shariah", "Social Science and Policy Studies Department, Worcester Polytechnic Tnstitute, Worcester, HA 61889" = "worcester Polytechnic Institut Tr “Dopartnen of Payenology, oshingtan and Lee"universiy, Lexsngton, vk 2488" = "agansngeon ana Lee Univesity" parent aycolopy, San Og State invari, fan leg, ch Botha" = “San Diego sate tniversi", Sl Niniverady or Fldrsea, Florida” = university of Flori Tr ‘University oF TIitwots st Urbana-Champaign, chanpag, TU° “University of Tinois at Urbara-chanpaany ‘University of Viegitio, vke'= "University of Virginio", . ‘arian Unversity, Indianapolis, 17 “narian Universi, ‘Departnet of Psyenolony, Lehace Collage, Tenacay AY 1489" ~ “Ithaca Col2ege", ‘Uhiversity of Michigan" = “university oF Michigan”, “opartnert af Payenology, Pennsylvania state University Abington, sbtagéon, “Breer of Paytaogy Tage aot usvesiy oleae Seton, 7740 ‘Department of Cognitive Sefence Occdental College, Los Angeles, CA" ~ “Ocetdental Coliege', ‘the aemsyivanta State University = "he Peonsplvania state University", oper University of Hong Kong” ~ “Open University of Hong Kong”, T MConcapeiny lle” = "Concepcion Alstiner (Loc, 120) soe Ox ‘linpse() BS tse ciety 2, 2,3, 4, 5, 6) 7, 8, 9, 18, 32, 22, 38, 24, 5, 16,37, 28, 29, 28, 22, 22, 23, 26, fe $ lts Cones Minerdsty oF Now we'll do some vicky wranaing with the ouput fom eaeft) and +x) to arrange th odds rato summaries for he population average and he location- specific result 4 this wilt help us format the Labels on the secondary yoonte storrat « “anction(ranber) { ‘ormase(runber, digits = oy format =“) > 1 grab the thet 5 sumaries foef(ect)soc[, , “Yoctorsicet/fect"] dsteverame() x mtsietloe 2 Gletinct(h, lee) 9% pulld)) 1X Urvange (Estimate) ‘axet (oe) teceeanet) stiee() 98 mtstetloe Binesrews(grous, average) 28 mutstctrane 2 ee stinate = explestinated, as. i: fe soins, bye Noe") BOR 2 Ser) 52 expt. 5)) x strangetrark) sitatet lobed ‘felse(ts;na(Lab), “POPULATION AVERAGE", 120), summary © traetayeforman(estinate), "[°y mysforaat(Q2S), "my format (O85), “1 ihinpse0) BS esttnece cal 13.092577, 5.96195, 7.261304, 7.986084, 7.940200, 7.991296, 6.269468, 8.995253, SES Erecteror Gb oves2s005, a acted, o.saantie, orsssedse, 83305724, 8.203679, 8.7844, 8.50 -ntpfsolomonkurz netify.applog/2020-10-16-bayesian-meta-analysis-mbrms-id 70‘1116123, 202 PM Bayesian meta-analysis in bemsll|A, Solomon Kure 2 $c2.5 cds 015705, 3.700289, 2.sez4er, 3.510026, 4.05748, 5.657872, 4.258794, 4.424 HS 7.5 Labty asivecon, 9%sersae, 34.2008, 1 o2ssbe, 35,257506, 12.51837, a6. 4u6862, 15. SES roe GOD on 73, 6, 6, GF, a, 8, 30, any fa) ah, ae, '36, S647, a8," 39, 28, 28,2 $ lnse1_ Sees “Paruuavion avemee, "Wank Inala workers", “Unvera\ty $ Stonary hes Saves [21re2, 45:78) "5.98 (3.78, 950)") 7026 (3.5 Here's our custom ores pla. {ues(x = tatinate, a jester ~ Tate © rer af) (aesisite #7 Estateror, color » option = °C begin = 33, end = 07) + (range = ey" 3-2)) * Sreake «i:6 © 30, expand « (mae = e(0.005, 0.09599) + breaks'= ors), Lupe = e(-1, oe), expene = lly Oy rvs, label) secrecy 2 fasbels = pat(erans, sumary))) + text = (Ganiiy's tines Sris-terty s (jut oy "eater = ses) ty-ripie = hsusee hy Ske 79, cece OF parei-sockaneun (e >, You may have noticed this plo is based on the rele of our mlilvel mode, we, We could ve done the same hase thing withthe rests fom the more conventional meta-analysis model, ne 00 [Pm not aware thie it typical in random effet meta-analyes, but it might be useful to father laf the meaning af the two primary parameters, ft and. Like with the forest plot, above, we could examine these with either nea oe. Foe kicks, Wel use 00 the coaveational Bayesian meta-analysis). In the output Fm ‘5 crows_de(me), J 9nd 7 ace the columns named _incerceat and so_20c_Entercept, respectively. dene i) {o_Inerceptist_toe_tatercapt) 0 hitpsisolomonkurz nett. apptblog/2020-10-16-bayestan-meta-analysis-r-brms-i!‘1116123, 202 PM Bayesian meta-analysis in bmsll|A. Solomon Kurz you sell back above, you'll see ou sino 8, which are themselves normaly distributed in he population of posible effec stes fom similar states: 8; ~ Normally, r). Why not we our postion to gets sense of what tis distribution might look lke? Recel that the parameter are i the logos metic. We'll ado effect meta-analysis explicitly presumed our empirical effectszeetimates y are approximations ofthe tue effect. samples to simulte draws from Normal( 1, ‘resent the disuibution in that mete and a odd ais. color & viesats pattoption = °CV07(6) seeesadis) SMontnete(Ls « enorm(a(), neon = B_Intercept, sf « sf Loe_intere (oF © Proraa(), wean = brintercept, Sd = sdrlocintercest) 39% 6009) OX sttce(ssie) mastainone ‘actor( nan to = “attact size") 0% abels = e( ogoets"y “odes ratto"))) 29% aplot(aestn = “effect siae'y y= 00) 4 Geansaste(color s cary fall © colony's Sey. continaous(@utt, breaks = MLL) + Liaetatecsiontvormal tegen tta})) + Thenettent = elemore-tent(finily = “ines"), iprbackaroubd = lever feet(colot'= “cansparent™)) + roptonane, ealea © fons") Both pals show 1,000 draws, each of whichis depicted by a single dt, f we were to run this experiment 1,00 ies and compute the eect ie sepatately foe ‘ach one, thi is what we'd expect thot distributions of effet sizes to ook like, Seems like here's lo of varaton inter, ck? The next ime you observe Your fellow scientists debating over whether a study replicated or not keep these distibution in mind. Once you start thinking sboutdistibutions, repletion becomes & tesky notion, Parting thoughts, Hl There are oter things you might do wit these data For example, you might inspect how much the effet size varies between thse ftom WEIRD and non-WEIRD than by cecation, Bu think We've gone far enough o get You sated, Details ‘= Mllevel models andthe indescwariable anpaoach Make model diagrams, Krschke sve» Oreactions © O comments Wete Preview ha Footer ©A, Solomon Kur (2022) Made with Hugo Apian ‘Based on Blovophoni by Fornsoe -ntpfsolomonkurz netify.applog/2020-10-16-bayesian-meta-analysis-mbrms-id s0110