Radically Transparent Research: Show Me The Codes

Mel Chua Readiness Assessment, Fall 2012

Prelude: How to read this work

This document is released under a Creative Commons Attribution Share-Alike 3.0 license. Feel free to remix and share as long as you credit this work and make your remixes publicly available under a similar license. Commentary is welcome at http://blog.melchua.com/2012/10/29/rat-the-document, the online home of this document.

Feel free to skip this prelude if you're not on my committee, interested in licensing, or simply want to jump into the nitty gritty of Radically Transparent Research already. For everyone else: this document was written in response to the three questions in my Readiness Assessment, the equivalent of qualifying examinations for Purdue's Engineering Education PhD program. You can find their full original phrasings in Appendix A at the document's online home.

You'll notice, however, that this document is a unified whole that doesn't quite answer them in sequence. That's why, if you're on my committee, you'll have gotten a copy of this PDF with colored bars running down the side. Red indicates this portion responds to the first question, yellow indicates a response to the second, and blue a response to the third.

If you'd like to examine some of the in-progress thoughts and notes that were taken in the process of writing this final document, look at my blog (http://blog.melchua.com) between the dates of October 15 and 29, 2012; a compendium of posts specifically related to this is also present at the end of the document as Appendix B at the document's online home.

Let's get started.

This paper attempts to:

Characterize radical transparency as an epistemological perspective and discuss what its patterns look like when applied to research this is an introduction to the concept of Radically Transparent Research (RTR). (background)

Discuss the affordances offered by RTR (Question #3, in blue) Characterize the nature of learning in radically transparent communities that results from taking advantages of those affordances (Question #1, in red)

Discuss how various qualitative research methods work with RTR (Question #2, in yellow)

What the heck is the title about?

It's a pun. In software development, coding is a basic act that refers to the creation of source material (code) that makes up the finished product. Show me the code is the open source software variant of talk is cheap, back it up with action. In qualitative research, coding is a basic act that refers to the categorization of source material; analysis of these codes eventually becomes the finished product, but the data, codes, and analysis themselves are typically hidden from view. Among other things, RTR requires that this material be exposed. Therefore, show me the codes is a directive that partially answers the question what does RTR do?

Introduction, or: I know this sounds crazy, but...

My first claim is an audacious one so audacious, in fact, that I've spent most of this writing period trying to prove myself wrong. I posit that radical transparency is an epistemological perspective

and that Radically Transparent Research (RTR) is emerging as a new qualitative research paradigm that is primarily about seeing and creating possibilities.

In order to do this, I need to explain what I mean by radical transparency and by research, since those are the two existing and separate components that RTR joins together.

What is Radical Transparency?

Radical transparency refers to the cultural practices used by healthy open communities (Free/Libre and open source software, open content, and open hardware projects) to expose their work in as close to realtime as possible and in a way that makes it possible for others to freely and nondestructively experiment with it.

Let's break this down.

...cultural practices...

Radical transparency is a praxis that has been adopted across various domains (for instance, the production of encyclopedias, operating systems, web browsers, and so forth) by communities of practice as part of their practice. A community of practice, a notion first articulated by sociologists Jean Lave and Etienne Wenger (1991), consists of a community of people who share a common practice within a domain of knowledge (Wenger, McDermott, & Snyder, 2002).

Exploring all the possible permutations for domains, communities, and practices gets complicated quickly. Multiple communities of practice may exist for a given domain; BSD, Linux, OSX, and Windows are all situated within the domain of operating systems development, but the former two

are radically transparent whereas the latter two do not. Not all communities that utilize radical transparency are successful, as Mako Hill's research (2012) on failed online enyclopedia projects shows (and plenty of communities without radical transparency do just fine, as Apple, Inc. demonstrates).

Furthermore, the use of radical trasparency as a praxis does not necessarily mean the presence of a community; Krishnamurthy's (2002) looked at the 100 most active mature projects on Sourceforge, a popular open source project host, and found the most common team size was... 1. To further complicate matters, not all projects that claim to be open actually implement radical transparency; Phil Marsosudiro coined the term fauxpen to refer to projects that talk the talk but fail to walk the walk (Searls, 2009).

...used by healthy open communities...

Before we run into a quandary of boundary articulation, I'll offer up a phrase specifically created to make it easier for us to discuss this phenomenon. Since the phrase radically transparent community of practice is somewhat awkward, we'll use the term open community as a shorthand to refer to any community that utilizes the praxis of radical transparency as part of their practice, regardless of how they self-identify. For our purposes, if it walks like a duck and swims like a duck and quacks like a duck, we're going to call it a duck (Heim, 2007).

Successful open communities are fascinating ducks. They're actually more like platypi strange hybrids that make researcher after researcher exclaim what the heck is that and how is it alive? What are the economic dynamics of these communities (Lerner & Tirole, 2002; Lakhani & Wolf, 2005; Lerner, Pathak, & Tirole, 2006)? Why would people join (von Krogh, Spaeth, & Lakhani, 2003), work in (Hars & Ou, 2002), and help colleagues in such a group? (Lakhani & Von Hippel,

2003) How can such an unorthodox arrangement possibly be successful? (Bonaccorsi & Rossi, 2003)

Against that sort of backdrop, one of the biggest early triumphs of open communities has simply been to prove the possibility of their existence to a sometimes-skeptical academic and corporate audience. Every successful project stands as a living, thriving example of what the dynamics and output of a radically transparent community in their domain could look like.

...to expose their work in as close to realtime as possible...

So what do these communities look like or more to the point, what exactly does this praxis of radical transparency entail? One key cultural can be summed up as if it ain't public, it don't count. Coleman's (2010) ethnography of hacker conferences described the validity and importance of such public discourse, noting that even small conversations at in-person gatherings would consistently reference and center around publicly-archived conversations.

A related pattern, release early, release often, ensures the frequent presence of a contributor's voice in these highly-valued public conversations; realtime exposure thus carries a clear benefit to the individual. Linus's Law, (Raymond, 2001) is sometimes cited as a benefit of this behavior to the larger community: enough eyeballs make all bugs shallow an argument fascinatingly similar to engineering education conversations about the importance of cross-disciplinary collaboration (Adams, Mann, Forin, & Jordan, 2009; Borrego & Newswander, 2008).

The desire to get one's work in front of more eyeballs motivates radical transparency practitioners to target each of their contributions towards the open community where it will benefit the most people, a pattern called pushing to upstream that inadvertently benefits component reuse and quality control

(Aberdour, 2007). The patterns of releasing early, releasing often, and pushing to upstream lead to the more eyeballs mentioned in Linus's Law, which is positively correlated with the rate of innovation adoption (Damanpour, 1991) regardless of the type of innovation or organization. Keeping up with this rapid development of new ideas and behaviors makes the realtime part of radical real-time transparency even more important to open community participants.

Ellis, Hislop, Chua, and Dziallas (2011) note that making something publicly and easily available does not necessarily imply widely advertising it, a distinction that preserves a reasonable signal-tonoise ratio. Multiple empirical studies by Elliott (2007) confirmed that when contributions are pushed upstream, the contributions themselves are used as stigmergic cues to negotiate contributions and the upstreams become workspaces that served as boundary objects that removed barriers to participation.

Participation is often solo and asynchronous, as Howison's ethnographies of variouso pen source projects (2008) revealed. However, they are done in company; Howison uses the phrase Alone Together as his title. Efimova's ethnography of blogging knowledge workers (2009) explored how these asychronous conversations became an indirectly enabling network that supported the sensemaking of participants who were able to embrace the transparency and fragmentation of [their] work that the process entailed.

The stigmergic nature of collaboration means that, at any given time, a contributor is likely to be creating a new whole by adding one small part to the top of an existing stack of components, leading to Robinson's Maxim of begin with the finishing touches (Karlie Robinson, personal communication, 2009).

This pattern is also a sign of respect for oneself and other contributors. As Eric Raymond's How to

Become a Hacker manifesto puts it, No problem should ever have to be solved twice... Creative brains are a valuable, limited resource. They shouldn't be wasted on re-inventing the wheel when there are so many fascinating new problems waiting out there... it's almost a moral duty for you to share information, solve problems and then give the solutions away just so other hackers can solve new problems instead of having to perpetually re-address old ones. (Raymond, 2012) Again, this stance of default to open (another common phrasing) is largely motivated by philosophy, with practical consequences following from this philosophical stance.

...makes it possible for others to freely...

I want to pause for a moment to discuss exactly what we mean by free. The philosophical roots of radical transparency can be traced back to the Free Software Definition, first written in 1986 by Richard Stallman, who continues to maintain it today (2012). As the title implies, the original document is about software, which does a user's computing and is composed of source code; I have adapted the text below to refer to artifacts that serve a purpose to the user and are made of components of some sort, so here are the Four Freedoms in their generalized form:

The freedom to use the artifacts for any purpose. The freedom to study how the artifacts work, and change it so it serves your purposes as you wish. Access to the components that comprise the artifact is a precondition for this.

The freedom to redistribute copies so you can help your neighbor. The freedom to distribute copies of your modified version to others. By doing this you can give the whole community a chance to benefit from your changes. Access to the components that comprise the artifact is a precondition for this.

Something that has the Four Freedoms is considered free. Something that does not have the Four

Freedoms is not considered free, regardless of what it does or does not cost the distinction between Libre and Gratis not being present in the English-language word.

...and non-destructively experiment with it.

Another pattern open communities follow is setting up the environment, artifact creation process, and social rules to get as close as possible to the ideal of cheap and reversible mistakes. In other words, they actually set themselves up to be low-risk, a characteristic correlated with high performance, which in turn is correlated with decentralization (Singh, 1986). A decentralized and low-risk environment enables the open door policy pattern, where membership and participation is as low-barrier as possible in terms of multiple factors, including resources, identity, and timing. The more you lower the cost of an unexpected experiment, the more generous you can be about who you allow in the door and who they're allowed to be once they are inside.

The community is thus primed for an attitude of opportunism and serendipity. Instead of a hardcoded end goal set at the start, decisions are driven by the people present and the resources and opportunities they bring to the table, a pattern found by Sarasvathy (2001) in the decisionmaking behaviors of repeatedly successful entrepreneurs. Other patterns in Sarasvathy's processes of effectuation included the assumption of dynamic, nonlinear, and ecological [as opposed to independent] environments and the practice of affordable loss, an expanded version of cheap and reversible mistakes.

What would the RTR community of practice look like?

The above question is another way to ask what it looks like to apply radical transparency to the

research world. As we've seen, a community of practice consists of a domain, a community, and a practice.

I think its important to note here that an individual RTR project is not a community of practice. Research projects can be circles, groups, teams, or classes. They certainly partake in a practice, and they may be situated within one or more communities of practice, but they are not communities of practice themselves. If that were the case, it would mean that the practice of that research project was unique to it, and the only purpose of the project was to further that practice. Participants may join a research project in order to learn the practice it utilizes (for instance, grounded theory analysis), but such projects are valuable learning experiences primarily because a community of practice exists beyond that specific project, and the skills they gain are thus transferrable to a broader world.

What could become a community of practice is RTR itself the practice of applying patterns of radical transparency to the domain of qualitative research. What prevents it from being a true community of practice right now is the lack of a real community; there are individuals implementing various radical transparency aspects in their research, and some of them even know each other, but they do not identify with a common cause... yet.

Domain: qualitative research

In this case, the domain is simple: qualitative research. Simply put, this is research that asks the kinds of questions that focus on... the intentions and perspectives of those involved in social interaction (Agee, 2009).

Since I am an engineering education researcher, I feel it is important to highlight that ...the

methods of educational research are often different from the methods of engineering research (Streveler & Smith, 2006, p. 104). Namely, engineering education frequently makes use of qualitative research methods, whereas engineering research is almost entirely qualitative.

This means that my primary field is caught between multiple scholarly domains that talk about and validate their work in different ways. In order for people in one domain to understand work being done in another, translation work must be done, and oftentimes it is the presence of a socially constructed power differential between the two domains that dictates the direction and manner of that translation (Becker, 2001, p. 328-329). In particular, the quantitative domain engineering research sits within is often regarded as harder and higher than the qualitative domain of disciplines like education research, with more perceived rigor and prestige. Oftentimes, it is the subordinate domains that need to explain themselves to the dominant ones, and I expect this paper will reflect that in its writing and tone.

Community: a few young researchers who haven't quite yet found each other

Radically transparent research refers to research done in a radically transparent manner rather than research done on something that is radically transparent. Radically transparent communities (open communities) can and have been studied without RTR, and RTR can be done on groups that aren't open communities.

That having been said, aspects of radical transparency patterns have been most often applied to research on open communities. In many cases, the adoption of radical transparency patterns came as a result of participatory research, as when Biella Coleman (2005) and James Howison (2008) became contributors Debian and BibDesk while they were studying those respective projects. Others became respected community members of a project long before entering academia to study

it, as with Mako Hill (2012) and Wikipedia and Martin Krafft (2010) and Debian. This prior experience shaped their personal research practices, as when Krafft exposed his data so others could reanalyze and validate it. Cormac Lawler (2011) actually created an open project, Wikiversity, in order to study it. Both Lawler and Mark Elliott (2007) engaged in action research by developing their work-in-progress on a wiki and inviting readers to edit it directly. They also wrote some public reflections on their process, but not nearly as extensively as Lilia Efimova, who liveblogged her reflexive auto-ethnography... on blogging (Efimova, 2009). Stian Haklev (2010) and Jason Priem (2010) advocate for open access and publicly release the tools they use for their own research... on open access.

I owe a huge debt to these researchers, who have used (and often pioneered) aspects of RTR in their own work. This is a recent and emerging trend; every single citation in the paragraph above is of that person's doctoral dissertation or a work in progress towards it.

Practice: radical transparency

When the practice or rather, praxis of radical transparency is implemented in qualitative research, here's what it looks like.

Pattern If it ain't public, it don't count .

Implementation All data is public (open data); closed data is never analyzed.

Examples Krafft released his dissertation data under a free license with permission from participants.

All publications are public (open access).

Every publication entry in Priem's CV is linked to the document, either self-archived or in an open access repository.

Authorship credit is determined None yet. entirely via public or publicly mentioned contributions. Release early, release often Intermediate results are released This paper, along with before the project is complete practically all of the ones memos, rough sketches, drafts, etc. mentioned above; the process of dialogue with study participants is an integral part of practically all qualitative research paradgims that came after post-positivism. Coleman and Efimova are particularly nice examples of this. Push to upstream Always push data, intermediate The pushing of final papers to analyses, and final papers to the the most public forum possible, most public forum possible. If fora are equally public, choose the one with the most prior work your work draws upon. and the one with the most prior work, is standard academic practice, albeit a largely tacit one. No similar practice has consistently been used for data and intermediate analyses. If you extend, modify, or critizue an idea, notify the originator or maintainer (intellectual heir) of that idea. This is hypothetically standard academic practice, but a very tacit one that typically falls under the category of networking. Begin with the finishing Use and cite existing work. All work mentioned above.

touches

This is standard academic practice, and a fairly explicit one, unlike others in this table.

Cheap and reversible mistakes

It should cost almost nothing to Lawler, Elliott, Krafft, and make a change in a local copy (and preferably the main copy as well). It should cost almost nothing to undo a change in a local copy (and preferably the main copy as well). While aliases and anonymity All work mentioned above. Haklev use(d) software for their dissertation data analysis that both allowed for public editing and had a single-click vandalism reversal feature.

may be allowed, nobody should This is standard academic be able to masquerade under someone else's identity. Optional: commit access to the All final scholarly outputs of practice.

main copy may be restricted, so the work mentioned above. The long as anyone can create and modify a local copy. intermediate outputs typically do not have such restrictions.

What does RTR on a non-open community look like?

All the research mentioned above looks at or works with a group that has always been an open community, which sometimes makes it difficult to discern which philosophies and patterns of transparency came about from the implementation of RTR in order to study the activity and which ones had already existed beforehand as part of the activity itself. There is no common terminology (the term RTR is my invention) and the exact nature of radical transparency has not previously been articulated this document is a first attempt to do so.

The Changemakers project is one example of such a research project. Changemakers (Adams et al., 2012) is a work-in-progress that draws on interviews with figures generally considered to be changemakers in their STEM disciplines in order to identify what their change knowledge is and how it was acquired. It doesn't study an open community. In fact, it doesn't even study a community; all the interviewees work in separate facilities ranging from academia to corporations to the government, and none of their employers are particularly strong drivers of transparency; if anything, they're drivers of confidentiality.

The changemakers do not comprise an open community, but the research project working to understand their insights and experiences is a space where radical transparency practices are utilized. Now, some of the changemakers participate in that space in various ways, and (while this hasn't happened yet) it's conceivable that they may someday encounter and collaborate with each other in that space. However, the starting grounds for that collaboration will be as contributors to the open research space, and any sort of radically transparent changemakers community that may evolve will grow from that the study isn't examining one to begin with.

Now, the Changemakers research team can be completely public on the open web, but sometimes the group under study doesn't have such an option. Is it possible to use RTR with things that must be kept confidential? For instance, a government or corporate social science research team might be working on a classified product or studying a group whose nature or existence can't be publicly revealed. Less dramatically, a college preparing for accreditation or a lab group preparing for publication may not want to expose all their work to the entire world just yet.

The answer is yes. When we use the word transparency, we always need to ask: transparent to whom? Even things put out on the open web aren't transparent to everyone, because not every

human being in the world has web access or is literate in the language we are writing in. In cases of transparency within confidentiality, we're simply drawing smaller and more explicit boundaries in terms of who will be included within the boundaries of our transparency practices.

Types of participants in RTR

Regardless of where those boundaries are drawn, we can picture the participants in RTR as follows:

Peripheral Expert: What's happening in the practice? Peripheral Novice: What is the practice?

Full expert: How is the practice seen?

Full novice: How is the practice done?

Both dimensions in the 2x2 matrix are oversimplifications that make me flinch. Reality is much more complex and fluid than this table indicates. A more accurate representation would display both novice/expert and peripheral/full as continuums, but even this misses out on the shifting, stuttering, and socially constructed nature of the terms. After all, the notion of novice and expert are relative, and Lave and Wenger (1991) point out that the concept of peripherality itself suggests multiple, varied... ways of being located in the fields of participation defined by a community, suggesting full participation as a better alternative to describing non-peripheral participants as standing in a single core or center that does not exist (p. 35-37).

I would much rather use dualities such as participation/reification (Wenger, 1998, p. 66) to describe participants and their activities, but realized that this would quickly render my writing unintelligible to all but the most stalwart academic. Wenger's definition of a duality will soon explain why: it is a ...single conceptual unit that is formed by two inseparable and mutually constitutive elements

whose inherent tension and complementarity give the concept richness and dynamism... two dimensions that interact [but] do not define a spectrum (p. 66). Perhaps I will return to this challenge at a later date for instance, in my dissertation. However, In the interests of keeping my writing (more) transparent to you, the reader, I have chosen to use simple dichotomies of opposites here so that we have a simple way to discuss the possibilities RTR enables for each of these four groups.

What are the learning possibilities enabled by RTR?

Now that we've established (temporarily and somewhat arbitrarily for convenience) four types of participants present in RTR, we are able to discuss the possibilities enabled by it using the concept of affordances. First articulated by James Gibson (1977) and developed further by Donald Norman (1988), an affordance is a combination of properties that a thing has that determine just how that thing could possibly be used (Gibson, 1977, p. 67; Norman, 1988, p. 19). In other words, as Norman puts it, affordances suggest what a thing "is-for." Gibson's original definition also specified that the combinations of properties that made up a particular affordance must be "taken with reference to" an actor, which was the reason we needed to define the groups of participants in the last section. The actors under consideration here are those peripheral/full/novice/expert participants involved in both the practice and research of college-level engineering education.

Our job here is a bit tricky, because we're describing affordances that most of the actors under question don't currently see. Norman gleefully describes the obviousness of affordances that are deliberately designed into existence, noting that they "provide strong clues to the operation of things. Plates are for pushing. Knobs are for turning. Slots are for inserting things into... the user knows what to do just by looking: no picture, label, or instruction is required." (Norman, 1988, p.

9). Alas, this is decidedly not the case with the RTR brings into our domain; since transparency is usually not something that's forethought in college-level engineering education, it's our job to decouple the notion of what-things-are-made-for (design intent) from what-they-can-be-used-for (affordance).

This act of repurposing is similar to Kuhn's (1962) notion of a paradigm shift, which is about looking at things that are already there in a different sort of way. Affordances are only easily perceived by those who grasp their meaning (Gibson, 1977, p. 67-69); those who don't simply perceive the same combinations of properties as noise (de Groot, 1965) because of their lack of context (Dreyfus & Dreyfus, 1980). However, Scarantino (2003) reassures us that "affordances are what they are independently of whether or not they are perceivable (some may not be), and independently of how they are eventually perceived (directly or indirectly)" (p. 954).

If we can see even the slightest edges of a few affordances, we can develop and clarify our perceptions simply by trying them out. Eleanor Gibson (wife to James) and Anne Pick describe a "perception-action reciprocity" in their book on infant development, wherein "perception guides action in accord with the environmental supports or impediments presented, and action in turn yields information for further guidance, resulting in a continuous perception-action cycle" (Gibson & Pick, 2003, p. 16). In other words, as you act, you get more information with which to act. Many affordances (including modifiability, forking, and more complex social and reciprocal affordances between actors) are offered by RTR, but we will focus on three here: perception, lurking, and failure.

The affordance of perception

...of and inside boundaries (for peripheral participants)

RTR certainly comes with several affordances that are obvious. For instance, the word "transparency" strongly suggests perception itself as an affordance. What sort of perception? Certainly one that goes beyond the ordinary ability to look at things; since the Four Freedoms give you access to the components that make up the things (Stallman, 2012), you also have the ability to look inside them. After all, if it ain't public, it don't count. We could describe one aspect of this affordance as perception-inside-boundaries, and it is an affordance of radical transparency taken with respect to actors outside those boundaries -- in other words, peripheral observers.

The idea of perception-inside-boundaries sits both in direct accordance with the ideal of scientific practice and in direct contrast to the reality of it. As Becker (2001) points out, "...sometimes [researchers] do [accord a special status to the knowledge created by research], treating a result as definitive and 'blackboxing' it" (p. 323). Fields have historically been protective and selfpreservational about their own boundaries. How do we tell us from not-us, things in our discipline from things outside it? For work that does fall within our discipline, what makes some "better" than others? You could call such a realization perception-of-boundaries, and it is also available to actors outside those boundaries.

...of and within boundaries (for full participants)

Perception-of-boundaries is also an affordance with respect to full participants situated within those boundaries. In Becker's view, the gatekeepers of boundary-making terminology such as "research" have a vested interest in "keeping unworthy pretenders from successfully appropriating [the term]" (p. 318). The field of engineering education research is fortunate to have gatekeepers who are cognizant of the power of boundaries to both illumniate and occlude, a sort of perception-withinboundaries.

One of the five interrelated strands of foundational engineering education research suggested by the National Engineering Education Research Colloquies in 2006 is the articulation of "what constitutes engineering thinking and knowledge within social contexts now and into the future" (p. 259), which I see as a usage of their power to assist with, rather than exclude the possibility of, perceptioninside-boundaries. This area of "engineering epistemologies" also includes an investigation of "the mechanisms by which these defining elements change over time," (p. 260) which can be thought of as paradigm-shifting, a highly-valued research activity. It thus becomes increasingly important to shift the idea of "paradigm shifting" itself and to bring ideas -- and people -- into "research" that were not inside its boundaries before (Nielsen, 1991, p. 23). Taking advantage of the affordance of perception allows a much larger "us" to transform current engineering education practice by defining what it is and who's "inside" it.

... across boundaries (for all participants)

Following the pattern of pushing to upstream reveals another type of perceptual affordance, that of perception-across-boundaries. When we push to upstream, we are really following evolving threads of questions and ideas as opposed to prescribed and socially-constructed boundaries. We thus often end up pushing ideas across those boundaries as a happy side effect, allowing more and more people to partake in a perception-across-boundaries of both our ideas and artifacts and the histories of the thoughts and components that compose them.

To give a concrete example, I am currently blogging on a forum frequented by hearing aid users with technological backgrounds. If I hear a question about hearing aid technologies on the forums, I try to get it answered in the audiology department, then follow the pattern of pushing to upstream by relaying the answers back to the forums. Through this engagement, the participating audiology

students come to see themselves as participating in "engineering education"; although they are not being trained as practicing engineers, questions and comments from the forums point out that they are certainly receiving training in engineering. The users in the forums also find themselves engaged in "engineering education" as contributors to the engineering training of the audiology students. Neither group's activities had previously been considered "engineering education" either by external observers or by themselves, but they are starting to be able to give that social construction of identity (Pawley, 2009) to each other, a gift enabled by their newfound ability to perceive the activities of the other group in even the limited and intermittent manner afforded by the liveblogging of a single overworked graduate student.

... as a force of legitimization and dissemination (for expert participants)

The affordance of perception, when taken in respect to experts, creates legitimization. The requirements for becoming an "official" engineering educator at the undergraduate level also high; instructors must survive both undergraduate and graduate school in engineering and prove their competence in research in order to be allowed to teach practice (Miller, 2011). Of course, there are many people in engineering communities of practice who act as mentors to novice engineers coming into their communities, but we generally don't recognize them as "real engineering educators," meaning that their efforts in educating engineers go uncounted. For that matter, the efforts of "official" engineering educators to educate engineers largely go uncounted, especially in top-tier institutions where evaluations for promotion and tenure hinge on research rather than teaching.

Since it has historically been engineering educators who enter engineering education research, we end up with a field where the average prior experience upon entry is undergraduate and graduate work in engineering followed by the development of skills in STEM research sufficient to earn a

faculty position (and possibly tenure) in an enginering department -- all prerequisites for many people before they're allowed to develop and utilize education research skills (which are vastly different from STEM research skills) with both safety and recognition. By this time it's often difficult to teach old dogs to do new tricks (Streveler & Smith, 2006). This is absurd, and I'm glad for the engineering education programs that have sprung up in the last decade in a direct attempt to counter that pattern.

When we engage in radically transparent research, we let it be known that "real engineering education researchers" are watching a space, which adds legitimacy to that space and the experts within it with very little additional time investment needed on their part. Furthermore, we make it increasingly possible for other experts with very little available time to exercise the affordance of perception and watch alongside us. When Borrego, Froyd, and Hall (2010) examined innovation diffusion in engineering education, they found word of mouth to be the most common way of spreading ideas about teaching among engineering department chairs; Fincher et al. (2012) also found this to be true for the spread of teaching practice transformations among engineering educators themselves.

If the sharing of stories is a primary way of disseminating practice, RTR's effects on legitimization and dissemination become even more pertinent. In many cases, RTR exposes the voices of people moving into a field, looking at and honoring the many diverse paths they may take into it. In fact, this was one of my driving forces in the development of what ended up becoming the paradigm; I have often taken unorthodox pathways out of neccessity: learning to play in a Madrigals recorder quartet where I couldn't hear the instruments of my fellow musicians, implementing conferencewide collaborative transcription systems so I could understand talks at events I was speaking at, concealing my age, gender, and race while doing technical work in and past college, pulling middleschool all-nighters hiding in the bathroom to read math books that weren't part of my allowed

curriculum. The non-standard routes I and others have taken are often invisible, unappreciated, and not prepared for, and we're often made to feel like we don't fit in the world and that we need to make up our own methods for anything we might want to do. However, when we start uncovering our stories, in addition to having our pasts seen, appreciated, and validated, we often learn that nonstandardization is more standard than we'd thought, and that there are methods and practices out there used by others that we can pick up and apply in our own lives as well.

If we further extend the affordance of perception to those without a lengthy list of expensive credentials, the combination of all these many types of watchers enables verification, legitimization, and dissemination of engineering education research both within and beyond the academy. The gap between engineering education research, engineering education practice, and engineering practice begins to close and blur -- and in fact, for the remainder of this paper, we will consider all three of these to be a combined domain joined by the affordance of perception-across-boundaries, as they have the potential to be with the addition of practices of transparency.

...of comprehensible conversation (for novice participants)

However, RTR isn't just about transparency; it's about realtime transparency. The realtime nature of this discourse exposure means that the discourse begins to resemble a conversational exchange rather than a series of unresponsive monologues. Since information is being released early and often, the stigmergic contributions described by Elliott (2007) build upon each other, and even isolated individuals working asynchronously (Howison, 2008) are able to participate in sensemaking collaborations (Efimova, 2009).

The speed of discourse has an interesting side effect on the nature of the language used within it. As an example, those who have been following my blogging on this Readiness Assessment may note

that my language in spontaneous blog posts is far more casual and less dense than the text that appears in this final and more scholarly document, though I am deliberately trying to keep it from becoming too mired in academic prose through the use of first-person speech and occasional asides like this one. Becker (2001) writes about the importance of using language that is "not just the high church version trotted out on formal occasions but the language of daily work as well," letting the ordinary people "who actually do the work" actually "speak for the science" (p. 323). It's not just conversation that the combination of transparent and realtime properties affords; it's comprehensibility. After all, more eyeballs only make bugs shallow if those eyeballs understand the conversation going on about the bug -- so the affordance of perception of comprehensible conversation becomes particularly salient taken in respect to novices in the domain.

When the conversation becomes radically transparent, we all become simultaneuosly "ordinary" people and "eminent" ones; we blend our languages and ways of speaking. In the course of this play, we become conscious of the languages we speak and which ones we honor above others in what circumstances. As in the example with audiology students mentioned in the previous section, we often end up validating common motion and common speech, making those who think they're in the formal "upper room" listen to voices that are usually invisible and swept under the rug, or as Mishler (1986) points out, modified in translation into a more scholarly format (p. 35-51). By enforcing the pattern of if it ain't public, it don't count, comprehensible conversations become perceivable by a broader spectrum of people who thus become potential legitimate participants in the engineering education community of practice as they gain knowledge of how to talk and be silent in the manner of full participants (Lave & Wenger, 1991, p. 105).

When I explain radical transparency to others, I frequently refer to the work of creative disciplines such as engineering as involving a rich and delightfully messy discourse situated in the midst of conversations between teammates and technology, components, codes, analysis, and constraints. I

also draw comparisons to language learning, noting that we generally recognize that memorizing vocabulary words or reading tourist guides will not give language students the ability to engage in "real conversation." And what is "real conversation?" It's a mess. The first time I visited Germany, I spent my long flights en route to Hannover by way of Chiang Mai (not a recommended itinerary, by the way) dutifully sticking the contents of an introductory grammar and phrasebook in my shortterm memory, only to discover that German families don't actually say the phrases found in the back of my pocket dictionary. We have slang, sentence fragments, backtracks, mistakes, and -- thankfully -- facial expressions and physical gestures, which were what ended up saving me. Linguistically, I was a helpless novice, rigidly dependent on taught rules and plans (Dreyfus & Dreyfus, 1980) that ended up not being a useful model of reality.

How many engineering students end up in the same situation? We already know that pre-college students have no idea what the field of engineering entails (National Academy of Engineering, 2008), but one would hope that students that elect to enroll in the field would be somewhat better off by the time they approach program completion. According to the Dreyfus model of skill acquisition (1980), my difficulty as a German-speaking novice was largely due to an inability to deal with context. This was forgivable; there's only so much that 20 hours of in-flight time with a book can do. However, engineering has historically been an intensive college major; back in 1918, Charles Mann pleaded for a reduction in courseload, noting that it was "absurd to require from the student more hours of intense mental labor than would be permitted him by law at the simplest manual labor" (p.257). Nearly a century later, the National Survey of Student Engagement (2011) still found engineering students to be hard workers, with 42% of seniors reporting that they spent over 20 hours per week preparing for class, the highest of any academic major (p. 16). Adding an extremely conservative estimate of 10 hours per week spent in engineering classes and multiplying across 8 semesters of 16 weeks apiece gives us 3,840 hours devoted to engineering learning, 192 times longer than my in-flight cram-fest. Furthermore, these hours often come with with scaffolding

and expert mentorship. If it takes 10,000 hours of deliberate practice to achieve expertise in a field (Ericson & Charness, 1994), we would expect at least some of those 42% of senior engineering undergraduates to be about 38.4% of the way there in their ability to deal with context.

That's why it's so disturbing that a longitudinal study described by Atman et al (2010) of 160 engineering undergraduates on four campuses over four years found that those four years did not increase students' abilities to consider broad context in design (p. 63) and that most were still uncertain about what it meant to be an engineer, even if 80% were planning to seek employment in the field (p. 71). I was struggling to understand spoken German because it was my first exposure to authentic conversations in that language. Could the difficulties of engineering students with openended, real-world problems in the workplace (Atman et al, 2010, p. 71-72) be due to the same thing -- a lack of exposure to "real engineering conversations" they can understand?

Access to comprehensible conversations has implications that go far behond the development of engineering disciplinary knowledge, which is only one of three components of engineering learning articulated by Stevens (2008). The other two, equally important, components of engineering learning are "engineering identity" and the navigational journey taken to develop it (p. 355-368). Mishler (1986) notes that when original "speech acts" are preserved in all their subtlety, informality, and messiness, observers are able to access vicarious experiences of "people like me doing that thing," which Bandura (1986) found was the second most influential factor in the development of self-efficacy, or the belief that "I myself could do that thing" (p. 399-408). Indeed, engineering identity -- whose development we briefly touched upon in the previous section on the affordance of perception -- is part of another major area of engineering education research identified by the National Engineering Education Research Colloquies in 1996, that of "engineering learning mechanisms," specifically "the learning progressions of learners and their educational experiences that develop this knowledge and identity necessary to be an engineer" (p. 260).

...of one's own actions in the form of modeling (for expert participants)

The comprehensible conversations overheard by novices are also actions of modeling being carried out by experts. Modeling is one technique of teaching within Cognitive Apprenticeship theory, which was developed by Collins, Brown, and Newman (1987) to extend the principles of traditional craft apprenticeships to cognitive fields such as engineering and research. Cognitive apprenticeship is not a model of teaching that gives teachers a packaged formula for instruction. Instead, it is an instructional paradigm for teaching (Collins et al., 1991, p. 17) that helps us understand how learning happens regardless of what teaching methodology is used (Lave and Wenger, 1991, p. 40), including "er, I wasn't even thinking of this as teaching in the first place."

Modeling involves the performance of a task by an expert such that novices who perceive it can build their own conceptual models of the processes required to accomplish that same task. In cognitive domains, this requires the externalization of usually internal processes and activities (Collins et al., 1991, p.13). Such an externalization is typically a separate process that experts need to be trained in, since they have since relinquished any reliance on rules and guidelines and have an intuitive grasp of how to navigate their context (Dreyfus & Dreyfus, 1980). However, in a radically transparent context, the pattern of if it ain't public it don't count means that all legitimate actions and their resulting artifacts are recorded examples of modeling that can be accessed in perpetuity.

When taken in respect to expert practitioners in a domain, the affordance of perceiving one's own expertise increases their ability to know and monitor their activities and that of others in the field. Their metacognitive skill increases as they become more conscious of learning processes in their domain, and experts come to percieve themselves as mentors and teachers as well as expert practitioners. Of course, metacognitive gains can be made by novices learning in the space as well

-- but in a different way, as we shall see shortly.

The affordance of lurking

... as a developer of metacognitive skills (for novice participants)

The affordance of lurking arises from the distinction between the affordance of perception and the affordance of participation; this affordance is a vital one and the reason it's called radically transparent research and not radically participatory research. Gibson (1977) uses an analogy to explain the difference: a glass pane affords perception but not [locomotive] participation, whereas a cloth curtain affords participation but not perception (p. 74). Watching television shows like Design Squad can be thought of as an engineering education example of the glass pane; you can see, but you can't touch. Filling out teaching evaluations for an engineering class is like the cloth curtain; you've thrown data into the aether, but have no idea if it will end up affecting anything.

When we separate perception from participation and incorporate the affordance of concealment, we open up the affordance of lurking, a form of concealment which may seem like the opposite of our first affordance of perception (Gibson, 1977, p. 73) but can also be described as a particular subvariant of it. When you are concealed, you do not offer the affordance of perception to others, but you can still exercise your ability to perceive them. We call this behavior lurking, and it is a valid activity in a community that practices radical transparency. Communities of practice need balance and interrelation between public and private spaces in order to be successful (Wenger, McDermott, & Snyder, 2002, p. 58-59).

Since cognitive apprenticeships work to "make thinking visible" so that novices "are able to see the process of work" (Collins et al., 1991, p. 1), novices are able to build their metacognitive skills by

observing experts build and articulate their own self-monitoring procedures. Through lurking, novices get a chance to build conceptual models of tasks and interpetative structures for making sense of the feedback they will get before attempting the tasks or receiving the feedback, which gives them an opportunity to develop their autonomous reflection skills (Collins et al., 1987, p. 5). They also learn "...who is involved [in a community of practice]; what they do; what everyday life is like; how masters talk, walk, work, and generally conduct their lives; how people who are not part of the community of practice interact with it; what other learners are doing; and what learners need to learn to become full practitioners. It includes an increasing understanding of how, when, and about what old-times collaborate, collude, and collide, and what they enjoy, dislike, respect, and admire" (Lave & Wenger, 1991, p. 95).

...as a cost reducer (for peripheral novice participants)

You may have noticed that all these benefits are available to novices engaging in full participation: those who don't lurk, those who participate as well as percieve, those who make themselves visible. So what's the big deal? The answer is that participation and visibility can be a costly endeavor, and if people need to participate and be perceived in order to percieve activity in a domain, they will sometimes opt not to watch at all.

Engineering is currently a field with a high entrance cost precisely because it does not separate the affordance of perception from the affordance of participation; we don't allow people to watch without "putting skin in the game," so to speak. If we take the undergraduate engineering experience as being the primary "game" to get "admission" to, an introductory-level ticket is 3,840 grueling hours of one's life plus far more dollars for tuition. With such a high initial time and monetary investment needed to "try out" the field, we shouldn't be surprised that very few people are particulary inclined to do so, especially with the additional emotional investment needed to cope

with the knowledge that "failure" means the high-visibility action of dropping out.

Some novices from underrepresented groups carry the additional cost of stereotype threat (Steele & Aronson, 1995), meaning that visibility of their group membership may call attention to the fact that they aren't "supposed" to do well, which is likely to make them actually perform less well. This cost doesn't even count the effort needed prior to college to meet engineering entrance requirements. The statistics for that are even more dismal; today's average high school graduate has chosen to take fewer than two credits of advanced math or science, and only 35% graduate having taken precalculus (National Science Board, 2012), making the "comprehensible" part of "comprehensible conversations" a very tricky one to pull off indeed.

Decoupling participation from perception and allowing for the usage of the affordance of concealment allows more people to afford the affordance of perception. At the same time, lurking is also an acknowledgement that perception is "an evolving form of membership" (Lave & Wenger, 1991, p. 53) and itself a form of participation. To understanding lurking as an affordance is to embrace a dialetical tension, a way of being in two contradictory places at once (Nielsen, 1991, p. 25-26) -- perception is both participation and separate from participation. Specifically, lurking is a form of legitimate peripheral participation, a term used to refer to participation in the "actual process of an expert [in a community of practice], but only to a limited degree and with limited responsibility for the ultimate product as a whole" (Lave & Wenger, 1991, p. 14). To be even more specific, lurking is the form of legitimate peripheral participation where the processes in question are metacognitive in nature and the responsibility for the ultimate product is nil. It is the lowestcost, lowest-risk way possible to engage with a domain.

...as a motivational factor (for novice participants)

Although Lave and Wenger (1991) did not use the term "lurking" in their work on cognitive apprenticeships, they nevertheless described the mechanism of lurking when they said that its purpose was "...not to learn from talk as a substitute for legitimate peripheral participation," but rather "to learn to talk as a key to legitimate peripheral participation" (p. 109).

Lurking helps new participants gain an ability to converse -- but it also helps them gain the desire to do so. When Edward Deci wrote a book (1996) summarizing his research with Richard Ryan on self-determination theory, he explained that autonomy is a key prerequisite for intrinsic motivation -- you can't be intrinsically motivated about something unless you feel like it's your choice to do it. Individual learners differ in the experiences they need in order to develop self-efficacy, or the belief that they can launch in and be successful; one person might need more vicarious experiences or verbal persuasion (pep talks) than another before they're able to launch into actual successful performance, the strongest self-efficacy builder of all (Bandura, 1986).

Where do these vicarious experiences and pep talks for encouraging learners come from? In cognitive apprenticeships like the ones present in radically transparent communities, they come from "access to a wide range of ongoing activity, old-timers, and other members of the community; and to information, resources, and opportunities for participation" (Lave & Wenger, 1991, p. 100101). A community of practice is full of practitioners at all different stages along the journey of learning, affording participants the opportunity to simultaneously be mentor and mentee in a complex web of interrelations (p. 56-57). "Since members are visible participants in the target skills... learners have continual access to models of expertise-in-use" (Collins et al., 1987, p. 5). Seeing multiple models of expertise and other learners at different stages in the process encourages students to see learning as a series of incremental stages (Collins et al., 1991, p. 2). This in turn can encourage them to develop a growth mindset, the notion that abilities can be developed through training as opposed to being fixed and innate (Dweck, 2006).

The affordance of lurking is a permanently present one; it is okay for participants to conceal themselves as long as they like, and to drop from visibility back into lurking at any point in time, and both experts and novices can move fluidly in and out of being in full or peripheral participation (Wenger, 1998). The existence of lurking as an affordance is one of the main factors that creates the safety cushion in the original comic image.

The affordance of messing up

The other affordance that creates the safety cushion is one of the more surprising ones offered by radical transparency: acceptable failure. This is a direct outgrowth of the pattern of cheap and reversible mistakes; the cheaper and more reversible failures are, the more acceptable they become. Individuals with a growth mindset are more likely to see failure as an opportunity for learning, and communities can nurture such a mindset in their participants by celebrating effort and learning from mistakes (Dweck, 2006). I am discussing this affordance with respect to all four quadrants of participation at once because cognitive apprenticeship theory celebrates the idea of novice-as-expert (Collins et al., 1991, p. 17). This is another dialectical tension that goes beyond the idea that "everyone is an expert at some things and a novice at others" to encompass concepts like the "expert novice." Diana Kimball (2012), who runs a consultancy by that name, describes the idea as follows: "Just as every good book needs a first reader, every project needs a first user... that perspective is what Expert Novice aims to offer."

Regardless of what your expertise is in (including novicehood), Collins et al. (1987) points out that mistake-making is part of the expert process, and that realistic depictions of expert performance must include "struggles, false starts, discouragement, and the like" (p.12). The affordance of perception is instrumental here as well in helping participants "realize that thrashing is neither

unique to them nor a sign of incompetence" (Collins et al., 1991, p. 11). So long as we "hold naive beliefs about the nature of expert [performance], thinking that [the task] is a smooth and easy process for 'good'[performers]" (Collins et al., 1991, p. 11), we will not allow ourselves to fail nor learn from failure: experts out of fear that they will no longer be considered experts, novices out of fear that they will never transition to becoming experts, peripheral participants out of fear that their participation will no longer be validated but instead dismissed.

What methods could be used with RTR?

If the affordances of RTR have the potential to generate fascinating activities, the next question we must ask as researchers is how we can best observe and analyze those activities. The easy answer is to say any of them -- it's a paradigm, after all, and paradigms do not dictate research methods, though they may have affinities towards particular methods more than others.

Regardless of what methods are used to gather and analyze data, RTR has two specific and crucial effects on the proceedings: one with regards to the nature of data, the other with regards to the roles of participants. I will discuss these first before launching into a discussion of specific methods that have an affinity for RTR: ethnography, active interviews, and grounded theory.

The boundaries between these categories are somewhat flexible and arbitrary, as we will see as we go along; for example, analysis occurs during collection and the activities of analysis also generate data, and it's hard to tell where group interviews end and ethnographies begin. As in the 2x2 table of novice/expert/full/peripheral participants earlier in this paper, we are using these categories for our semantic convenience and to make it easier to discuss these ideas in a reasonable manner.

The effect of RTR on the nature of data, and its limitations We have already discussed Mishler's (1986) caution against the unconscious distortion of speech acts that happens in typical interviews, where a rich range of data (tone of voice, facial expression, and so forth) is available during face-to-face capture, but only some aspects (audio) are recorded during the actual conversation, and then further subtle details are lost during transcription and analysis in exchange for a gain in focus and clarity. The same data loss tradeoffs occur in other methods of data collection such as ethnography. RTR sets the stage for a different sort of tradeoff; the if it ain't public it don't count pattern only begins seeing things as data once they are captured in a publicly reproducible and transmissible format, whether that be in text, audio, video, or some other means. Once captured, the implementation of cheap and reversible mistakes leads to an adamant guard against permanent loss of any subtle details in the data. Although individual products stemming from the data notes, papers, and so forth will often choose to focus on some details to the exclusion of others, the full versions of the original data remain available in historical archives for future users.

To some extent, this can be seen as a critical theory move to equalize access and power between research participants as much as possible so that no one group is privileged over another for analysis, with the recognition that perfect achievement of this is virtually impossible in practice. In fact, the RTR tradeoff brings limitations and exclusions of its own.

First, individual participants are assumed to be capable of speaking reliably and validly (Holstein & Gubrium, 1995, p. 74) and of being mentally, legally, and logistically able to collaborate and enter into work relationships with others, which limits full participation to cognitively capable adults with access to the forum of engagement and the tools needed to participate in it. The chosen forum is often (but does not have to be) online, which excludes the millions of people without computer or internet access, and millions more for whom this access is too difficult or infrequent to

keep up with the fast pace of a distributed community. If the chosen forum is not online, access to the physical space either through colocation or some form of telepresence becomes a requirement instead. Although those excluded (including children) can enter into forms of legitimate peripheral participation with the aid of helpers and scaffolding, this is not the same as having the option to move into independent full participation.

Additionally, some datasets simply should not become public and identifiable except, of course, in cases of extraordinary (and most likely subject-initiated) consent -- medical, school, and work records, conversations in counseling and sensitive situations between family and friends, and other areas typically overseen by Institutional Review Boards (IRBs) that monitor research ethics for an academic institution. IRBs have a purpose, and my intent with RTR is not to call for their dissolution, but rather to suggest an alternative way of thinking about privacy, anonymity, and consent in research.

The effect of RTR on the roles of participants

Mishler (1986, p. 122-132) has a particularly keen examination of how researcher-subject roles have evolved over time, with a critical theory examination of the power differentials present in each incarnation of the relationship. The first and simplest relationship was researcher/responder; moving beyond this post-positivistic stimulus-response perspective led to the relationship of reporter/informant, where the people engaged in dialogue begin to treat each other as significant others rather than objective scientific objects. However, the balance of power still lies in favor of the reporter, who gets to choose the filters used in his or her report. An alternative view of the relationship is that of learner-actor/advocate, which still retains the view of learner-actors as people who must be advocated for, carrying implicit undertones of privilege of access.

The paradigm of RTR treats people we traditionally call interviewees as research collaborators, full participants in the development of the study and in the analysis and interpretation of data (Mishler, 1995, p. 126). In effect, the distinction between researcher and subject is deliberately thrown out we are all participants with various levels of expertise and participation, and the activities we partake in are simultaneously the activity under study and the study of that activity. This gives an amusing twist to Becker's (2001) statement of what he describes as Bruno Latour's rule of method: ...we should be as undecided as the actors we study. If they think a conclusion, a finding, or a theory is shaky, controversial, or open to question, then we should too. And we should do that even if what we are studying is a historical controversy whose outcome we now know, even through the actors involved at the time couldnt (p. 323). In RTR, we are the actors we study. If we think a conclusion, a finding, or a theory is shaky, then... by the reflexive property, of course we do! As part of their role as research collaborators, participants in an RTR project are identified; after all, collaborators must be given credit for their work. Identifiers do not necessarily mean real names; stable aliases are an equally legitimate alternative that allow participants to build relationships with each other. Contrast this to conventional research practice where the interviewee is typically anonymized, ostensibly for protection. This protection ensures that ...they will be treated as part of an anonymous mass... they will not be held personally responsible for what they say, nor will they be credited as individuals for what they say and think. In brief, they are deprived of their own voices (Mishler, 1986, p. 125). Thinking of participants as research collaborators rather than subjects in a separate category from oneself makes it far more difficult to deny their voices. The notion of interviewees as research collaborators is not new, but is usually only implemented partway, as evidenced by the presence of a separate category of interviewees joined to the phrase ...as research collaborators. Mishler (1995) describes what a typical partial implementation looks like: ...respondents' views may be presented, usually as a postscript, but the research aims and methods, as well as lines of analysis and interpreation, remain very much under the control of the

investigator (p. 127). In this way, a partial implementation retains the traditional subject/researcher power imbalance, particularly in regard to the timing, venue, and nature of the final output. In contrast, a full implementation of RTR results in a decentralization of open artifacts that decouples the questions of final output from the initiating researcher in theory, any participant in the project can take and present the work at any venue, so long as they appropriately credit the contributions of others. This possibility is afforded by conventional research practice, of course but it is a hidden one that often remains undiscussed, perhaps because of the extraordinary amount of work it would take to obtain the necessary data and permissions, and perhaps because it is thought that the originating researchers will want academic credit for their work and will therefore frown upon such an action. An RTR project makes this particular affordance of forkability more perceptible and explicit, encouraging it as a way to get academic credit in the form of citations rather than publications.

A brief reminder

In the sections that follow, I encourage you to keep in mind the thought the distinctions can be blurred that's not the only way of looking at the world! whenever you encounter the following words presented as separate categories: data generation/collection/analysis subject/researcher, participant/researcher, interviewer/interviewee, etc.

The use of ethnography in RTR

We now turn our attention to specific research methods that share an affinity with RTR. Perhaps the simplest RTR experimental design is an ethnographic one, where the researcher's objective is describing, analyzing, and interpreting a culture-sharing groups shared patterns of behavior,

beliefs, and language that develop over time (Creswell, 2008, p. 473). It has certainly been applied by multiple researchers (Coleman, 2005; Efimova, 2009) using aspects of radical transparency in their research. However, it does have its limitations. Creswell continues: You conduct an ethnography when the study of a group provides understanding of a larger issue. You also conduct an ethnography when you have a culture-sharing group to study one that has been together for some time and has developed shared values, beliefs, and language. You capture the rules of behavior (p. 473) An interesting aspect of RTR is how many of its affordances are dynamic as opposed to static in nature. Perception and lurking feel more static, and as such are well-suited to being captured by ethnographic techniques as well as being used themselves during an ethnographic study. In contrast, failure, modifiability, and various forms of social reciprocity and learning speak to moving and unstable boundaries where multiple cultures are blending and mixing. It's not simply the fact that novices are being initiated into a system that creates this dynamicness; when the rules and processes of initiation are set, ethnographic techniques are still appropriate. However, when the practice or domain they are being initiated into is itself topsy-turvy and shared values, beliefs, and languages are still emerging, it is impossible to write about the way things are because they aren't there yet. One question that determines the appropriateness of ethnography to an RTR research design is the nature of the community under study. Does the community already practice radical transparency, as in Coleman's (2005) study of Debian and Efimova's (2009) study of blogging in knowledge workers? If the various patterns of radical transparency are already being consistently implemented, ethnography can be employed to uncover them and trace their implications. Ethnography becomes about setting the stage and knowing and being able to articulate the ground rules that keep the space safe.

When the community does not yet practice radical transparency, RTR can be thought of as an intervention in the space, even if the community itself does not become an open community in the process. In this case, RTR itself becomes a monkey wrench that makes the boundaries unstable, rendering ethnography a less appropriate technique. However, if the community or communities in question have existing stable practices that will be disrupted, ethnography can still be used in these cases for groundwork prior to RTR's usage as an intervention in order to characterize their starting state. If were looking at creating a ratatouille of woodworking and yoga, we can look at ethnographies of woodworkers and yogis as conversational artifacts to help us understand the spaces our various participants might be coming from and to help us unpack the narratives of their experiences as they report them. Adding active interviews to the mix

This implies that another method for RTR is interviewing. I will focus here on a technique called active interviewing as described by Holstein and Gubrium (1995), although interviews themselves are a broad category of data collection that encompass even highly structured interactions such as the Delphi method used by Krafft (2010). My choice to concentrate on active interviews is a response to Mishler's (1986) critique of conventional interview practice as having a behavorial bias that positions questions as stimulus-response engagements that can be taken independently of each other (p. 9-34). As defined by Holstein and Gubrium, active interviews bring ...meaning and its construction to the foreground. An active approach might therefore be most appropriate in those instances when the researcher is interested in subjective interpretations, or the process of interpretation more generally, even for ostensibly well-defined information (1995, p. 73). The hint about ostensibly well-defined information suggests a pairing of active interviewing with ethnographic techniques, a combination Holstein and Gubrium advocate for elsewhere in their book (p. 45). At the simplest level, we might use ethnography as a way to spark the unspinning of narratives in interviews by generating common

artifacts that conversations can center around, open datasets that participants can analyze from different viewpoints. In other words, one way of looking at ethnographic work in RTR is that it creates and characterized the space where perception can be afforded in the course of active interview conversations. In the context of RTR, the affordance of perception is a forcing function that can cause participants to simultaneously assume the role of cognitive master exposing their own thinking in actions of modeling as well as the role of cognitive apprentice viewing the exposed and modelled thinking of others, and this activity sometimes reveals the ill-defined nature of things we once thought to be well-defined. Mishler (1986) points out that allowing this discourse helps us avoid the trap of assuming standard and shared meanings, which allows meaning to emerge through and be realized in the discourse itself (p. 65). Another motivation for combining ethnography and active interviewing is to build the background knowledge of the active interviewer, who is not the uninvolved blank slate criticized by Mishler (1986) as attempting to provide solely behavioralist stimuli. Holstein & Gubrium point out that such attempts are futile; interviewers have background knowledge and prior relationships that will cause interviewees to respond to them in particular ways (1995, p. 38-45), and they are involved in the local inventiveness of the interview process (p. 70). Instead, the personhood and history of the interviewer is to be embraced; qualitative researchers are instruments, and as such, we must understand our changing selves. RTR participants have different backgrounds and can thus elicit different perspectives and stocks of knowledge from other participants. The background knowledge of an RTR participant also changes and grows as they continue to participate in the study; relationships change and shared events become things that conversations can rely on as a common reference point. The Sage2Guide study is an example of an RTR project that takes advantage of this; reading transcripts of the interviews with faculty members soon reveals that conversations are referencing past interviews, including dialogues with other participants not present (Chua & Dziallas, 2012), actions made possible by the

open licensed nature of the data. In this way, participants who are not synchronously present in the same room are still asynchronously brought into dialogue with each other, while a growing stockpile of background knowledge makes it possible to pose concrete questions and explore facts of respondents' circumstances that would not otherwise be probed, in contrast to conventional interview practices, which seek to limit informational spillage between interviews (Holstein & Gubrium, 2005, p. 46). If the aim is not to search for the best or most authentic answer, but rather to systematically activate applicable ways of knowing the possible answers... as diverse and contradictory as they might be (Holstein & Gubrium, 1995, p. 37), spillage is a resource to be celebrated and taken advantage of. Spillage can occur across interviews; it can also occur across participants. If, as previously discussed, RTR discards the conventional categories of subject and researcher in favor of talking about various forms and levels of participation, why should one group of participants do all the interviewing and another group always be the interviewed when so many rich combinations of person-to-person interactions are possible? This is a shift in viewing existing activity that blurs the distinction between ethnography and interviewing. If we take the usual ethnographic setup to consist of researchers observing interactions between subjects, then remove the researcher/subject distinction, we end up with some intriguing transformations. For instance, subject-subject interactions are seen to simultaneously be subject-researcher and researcher-researcher interactions. This turns all RTR engagements into multivocal interviews, where more than one interviewer and/or one interviewee are present in a narrative sense (Holstein & Gubrium, 1995, p. 66). Note that the reflection of a single individual can be a multivocal interview when that individual takes on multiple narrative roles during the course of their monologue. By taking advantage of the existence of reciprocal affordances between the narrative roles of participants, multivocal interviews can elicit different stocks of knowledge. Also note that, just as the presence of only one participating individual in an interaction does not exclude the possibility of multivocality in that

interaction, the presence of multiple individuals (a group interview) does not automatically create multivocality. The presence of others playing different roles in a conversational space can help affect and shape participant behaviors including causing them to fall silent (p. 66). This is why the presence of both public and private spaces (Wenger, McDermott, & Snyder, 2002, p. 58-59) and the affordance of lurking is so vital. The same dissolution of subject/researcher roles that create multivocality also turn data collection and analysis into the same process, a joining that has important implications for the use of grounded theory, as we will see shortly. Let us return to our example of the subject-subject interaction seen in a conventional ethnographic setup. We discussed how, with RTR, we can see the subject-subject interactions as simultaneously being subject-researcher and researcher-interactions. The former interaction could well be described as an interview, and the latter as a data analysis and sensemaking event so an interview, an ethnography, and a data analysis are present in the same event. Holstein and Gubrium call this "indigenous coding" (1995, p. 56), a term that refers to the natural conversational process of trying to describe and clarify what you've already said. Phrases like "you mean..." or "to sum it up," or "one way of looking at it..." are usually signals of indigenous coding. The role of the interviewer or in the case of RTR, the role of any participant in an engagement is to be a catalyst. Holstein and Gubrium are adamant that active interviews have an underlying organization despite their flexibility; in order to create a good meaning-making occasion, participants must be prepared to furnish precedence, incitement, restraint, and perspective as the interview proceeds, not to avoid them. In this way, ..through repeated reformulations of questions and responses, [participants] strive to arrive together at meanings that both can understand (Mishler, 1986, p. 65). Adding grounded theory to the mix

Grounded theory methods are ...a logically consistent set of data collection and analysis procedures aimed to develop theory... [that] consist of a set of inductive strategies for analyzing data (Charmaz, 2001, p. 335). Instead of beginning with theories and pulling data from a representative sample to fill in the boxes they've set up, grounded theory researchers begin by working through individual incidents in an area of study, allowing their codes to emerge from the data and their theories to emerge from the codes. Charmaz (2001, p. 336) describes 6 distinguishing characteristics of the methodology:

1. Simultaneous data collection and analysis 2. Codes come from data, not preconceived hypotheses 3. The presence of middle-range theories 4. The presence of memos as an intermediary between codes and final papers 5. Theoretical sampling for theory construction rather than representational sampling 6. The literature review is delayed until the end

We will examine each of these in turn as they apply to RTR and the intertwined methods of ethnography and active interviewing that we have already discussed as methods of data collection.

The first characteristic of grounded theory, which positions data collection and analysis as simultaneous; in practice, this means that the researchers (as separate from the subjects) are conducting both processes in parallel. However, as mentioned earlier, the blending of subject/researcher roles in RTR-style active interviews turn data collection and analysis into not just simultaneous processes, but the same process through the recognition of indigenous coding.

The second characteristic of grounded theory, the practice of allowing codes to emerge from the data rather than be set at the start by existing frameworks, has two important affinities with RTR.

First, it ...force[s] the researcher to attend closely to what happens in the empirical world he or she studies (Charmaz, 2001, p. 337), a level of presence and participation that facilitates the building of shared background contexts and relationships celebrated in active interviewing (Holstein & Gubrium, 1995, p. 38-45). Grounded theory captures process, so when it is applied within an RTR paradigm, it captures the process we are participating in, facilitating more metacognition and reflection on our own actions as discussed in the earlier section on the affordance of perception.

The second affinity has to do with access in this case, access to existing theoretical frameworks. Adademic theories are, by and large, quiet and rare things unencountered by the majority of people on this planet. Therefore, bringing in academic theories and categories at the start of a research project puts up several barriers to entry: first, the task of accessing the materials describing those theories no small feat, given that approximately 80% of scholarly articles still require expensive subscriptions or access to university resources, even in recent years when the push for open access has been especially successful (Bjork et al, 2010). If the materials can be accessed, they must still be understood, another nontrivial and time-consuming feat, since academic papers can sometimes be written in obfuscated prose and are often heavily reliant on citations of additional papers that are also likely to be densely written and closed-access. (My apologies to the readers of this paper for my sins in this regard.) To top it all off, after wresting with the twin demons of access-to-materials and access-to-understanding, the exhausted participant may reach the summit of literature only to find that the framework does not do a particularly accurate job of characterizing the emergent data.

However, in RTR, the emergent data itself is openly available and far more evenly accessible to a broad range of participants, as we have discussed so why not begin from that? Since data is being generated from experiences happening in realtime, it is equally new to everyone at the time of its emergence. The open dataset becomes a common experience and a shared artifact that participants can discuss, and those participants with knowledge of theory can bring that knowledge to the data,

rather than stuffing the data into the theory. The grounded theory characteristics of middle-range theories and memos thus becomes a communal activity as participants move, both individually and together, from line-by-line coding to focused coding using frequently occurring codes (Charmaz, 2001, p. 341-346).

The grounded theory technique of memo-writing fits in particularly well with practices of radical transparency, and can also be framed as the activity of making thinking visible so vital in cognitive apprenticeships (Collins et al., 1991). Grounded theory explicitly frames memo-writing as an experimental, exploratory space between codes and final papers where half-baked ideas can be tried out and revised (Charmaz, 2006, p. 72-73). My blog posts during the process of writing this Readiness can be thought of as memos in this sense; as Charmaz (2001) recommended, I have ended up weaving my blog memos into the finished product (p. 347-351), although the term finished is mildly misleading, as RTR documents are in a perpetual process of evolution as they constantly absorb the outputs of release early, release often; they're never done, they're merely shipped.

Grounded theory's usage of theoretical rather than representational sampling fits nicely with the subject selection process of active interviews, which Holstein and Gubrium (1995) describe as emergent and ongoing; any initial guesses at who the final group of participants will be are necessarily tentative, provisional (p. 74-75), and probably wrong. Charmaz (2001) describes this process as following the interesting questions (p. 337, p. 351) in order to get the data that will illuminate your emerging theory (p. 338-339) a pragmatic pursuit of understanding that constrasts with conventional notions of validity and representation in sampling. If our purpose is to develop a theoretical analysis that fits the data and has relevance to the area of study (p. 351), we must aim to cultivate a garden-bed of relevant data with sufficient richness and density for theory to sprout from.

Afterword

I don't quite consider this paper to be complete. Throughout the document, you've likely noticed me struggling to define, refine, and establish melting boundaries while talking about RTR; I'm showing a static picture while attempting to describe a movie.

There is one gigantic missing spot in this paper, and it's a hole I hope to fill during my time in Ohio next semester; I have not yet been able to characterize RTR as a paradigm or to situate its evolution alongside other qualitative paradigms (post-positivism, constructivism, post-structuralism, etc), largely because I don't yet understand how paradigms evolve and are characterized.

References
Aberdour, M. (2007). Achieving Quality in Open Source Software. IEEE Software, 24(1), 5865. Adams, R. S., Chua, M., Denick, D., Mondisa, J.-L., Sambamurthy, N., Siddiqui, J., Vanasupa, L., et al. (2012). Work in Progress: In their own words - how changemakers talk about change. Proceedings of the Frontiers in Education Conference. Presented at the Frontiers in Education, Seattle, WA: IEEE. Adams, R. S., Mann, L. M., Forin, T., & Jordan, S. (2009). Cross-Disciplinary practice in engineering contexts. Proceedings of the International Conference on Engineering Design. Presented at the International Conference on Engineering Design, Stanford University. Agee, J. (2009, July 9). Developing qualitative research questions: a reflective process. International Journal of Qualitative Studies in Education. Atman, C. J., Sheppard, S. D., Turns, J., Adams, R. S., Fleming, L. N., Stevens, R., Streveler, R. A., et al. (2010). Enabling Engineering Student Success: The Final Report for the Center for the Advancement of Engineering Education. Morgan & Claypool Publishers. Bandura, A. (1986). Social foundations of thought and action: A social cognitive theory. Englewood Cliffs, NJ: Prentice-Hall. Becker, H. S. (2001). The Epistemology of Qualitative Research. In Emerson R. (Ed). Contemporary field research: perspectives and formulations. (317-334). Prospect Heights, Ill: Waveland Press. Bonaccorsi, A., & Rossi, C. (2003). Why Open Source software can succeed. Research Policy, 32(7), 12431258. doi:10.1016/S0048-7333(03)00051-9 Borrego, M., Froyd, J. E., & Hall, T. S. (2010). Diffusion of Engineering Education Innovations: A Survey of Awareness and Adoption Rates in U.S. Engineering Departments. Journal of Engineering Education, 99(3), 185207. Borrego, M., & Newswander, L. K. (2008). Characteristics of Successful Cross-disciplinary Engineering Education Collaborations. Journal of Engineering Education, 97(2), 123134. Charmaz, K. (2001). Grounded Theory. In Emerson R. (Ed). Contemporary field research: perspectives and formulations. (335-352). Prospect Heights, Ill: Waveland Press. Coleman, G. (2005). The social construction of freedom in free and open source software: Hackers, ethics, and the liberal tradition. University of Chicago. Coleman, G. (2010). Hacking In-Person: The Ritual Character of Conferences and the Distillation of a Life-World. Anthropological Quarterly, 83(1), 4772. Collins, A., Brown, J. S., & Holum, A. (1991). Cognitive apprenticeship: making thinking visible. AMERICAN EDUCATOR, 6, 3846. Collins, A., Brown, J. S., & Newman, S. E. (1987). Cognitive Apprenticeship: teaching the craft of reading, writing, and mathematics (Research No. 403). Center for the Study of Reading Technical Reports (pp. 137). Champaign, IL: University of Illinois at Urbana-Champaign. Damanpour, F. (1991). Organizational Innovation: A Meta-Analysis of Effects of Determinants and Moderators. The Academy of Management Journal, 34(3), 555590. Dreyfus, S. E., & Dreyfus, H. L. (1980, February). A Five-Stage Model of the Mental Activities Involved in Directed Skill Acquisition. Storming Media. Dweck, C. S. (2012). Mindset the new psychology of success. Random House Inc. Groot, A. D. de. (2008). Thought and choice in chess. Amsterdam: Amsterdam University Press. Efimova, L. (2009). Passion at work: blogging practices of knowledge workers. Novay, Enschede.

Elliott, M. (2007). Stigmergic Collaboration: A Theoretical Framework for Mass Collaboration. University of Melbourne. Ellis, H. J. C., Hislop, G. W., Chua, M., & Dziallas, S. (2012). How to Involve Students in FOSS Projects. Proceedings of the Frontiers in Education Conference. Presented at the Frontiers in Education, Rapid City, SD: IEEE. Engineering Education Research Colloquies. (2006). The Research Agenda for the New Discipline of Engineering Education. Journal of Engineering Education, 95(4), 259261. Ericsson, K. A., & Charness, N. (1994). Expert performance: Its structure and acquisition. American Psychologist, 49(8), 725747. doi:10.1037/0003-066X.49.8.725 Fincher, S., Richards, B., Finlay, J., Sharp, H., & Falconer, I. (2012). Stories of Change: How Educators Change Their Practice. Proceedings of the Frontiers in Education Conference. Presented at the Frontiers in Education, Seattle, WA: IEEE. Gibson, E. J., & Pick, A. D. (2003). An ecological approach to perceptual learning and development. Retrieved October 17, 2012, from http://site.ebrary.com/id/10269203 Gibson, J. (1977). The Theory of Affordances. In Shaw, R. & Bransford, J. (Eds). Perceiving, acting, and knowing: toward an ecological psychology. (67-82). Lawrence Erlbaum Associates; distributed by the Halsted Press Division, Wiley. Haklev, S. (2010). The Chinese National Top Level Courses Project: Using Open Educational Resources to Promote Quality in Undergraduate Teaching. University of Toronto. Hars, A., & Ou, S. (2002). Working for free? Motivations of participating in open source projects. International Journal of Electronic Commerce, 6(3), 2539. Heim, M. (2007). Exploring Indiana highways: trip trivia. Wabasha, Minn.: T.O.N.E. Pub. Hill, B. M. (2012). Almost Wikipedia: What eight early online collaborative encyclopedia projects reveal about the mechanisms of collective action [Abstract]. Retrieved from http://cyber.law.harvard.edu/events/luncheon/2011/10/makohill Howison, J. (2008, December). Alone Together: A socio-technical theory of motivation, coordination and collaboration technologies in organizing for free and open source software development. Syracuse University. Kimball, D. (2012). Expert Novice. Retrieved from http://expertnovice.com Krafft, M. (2010). A Delphi study of the influences on innovation adoption and process evolution in a large open-source project the case of Debian. University of Limerick. Krishnamurthy, S. (2002). Cave or Community?: An Empirical Examination of 100 Mature Open Source Projects. First Monday, 7(6). Retrieved from http://firstmonday.org/issues/issue7_6/krishnamurthy/index.html Kuhn, T. S., & Hacking, I. (2012). The structure of scientific revolutions. Chicago; London: The University of Chicago Press. Lakhani, K., & Wolf, R. G. (2003). Why Hackers Do What They Do: Understanding Motivation and Effort in Free/Open Source Software Projects. SSRN Electronic Journal. doi:10.2139/ssrn.443040 Lave, J., & Wenger, E. (1991). Situated learning: legitimate peripheral participation. Cambridge [England]; New York: Cambridge University Press. Lawler, C. (2011). Edit this space: participation and expansive learning in developing Wikiversity. University of Manchester. Lerner, J., Pathak, P., & Tirole, J. (2006). The Dynamics of Open Source Contributors. American Economic Review, 96(2), 114118. Lerner, J., & Tirole, J. (2003). Some Simple Economics of Open Source. The Journal of Industrial Economics, 50(2),

197234. doi:10.1111/1467-6451.00174 Mann, C. R. (1918). A Study of Engineering Education. The Carnegie Foundation for the Advancement of Teaching. Miller, R. K. (2011, November). The Challenge of Spreading Innovation in Teaching and Learning. Retrieved from http://www.olin.edu/about_olin/pdfs/The%20Challenge%20of%20Spreading%20Innovation%20-%20Nov%202011.pdf Mishler, E. G. (1986). Research interviewing: context and narrative. Cambridge, Mass.: Harvard University Press. National Academy of Engineering. Committee on Public Understanding of Engineering Messages. (2008). Changing the conversation: messages for improving public understanding of engineering. Washington, D.C.: National Academies Press. National Science Board (2012). Science and Engineering Indicators 2012. ed. Arlington VA: National Science Foundation National Survey of Student Engagement. (2011). Fostering student engagement campuswideannual results 2011. Indiana University Center for Postsecondary Research. Nielsen, J. M. (1991). Introduction. Feminist Research Methods: Exemplary Readings in the Social Sciences (pp. 132). Boulder: Westview Press. Norman, D. A. (1988). The psychology of everyday things. [New York]: Basic Books. Pawley, A. L. (2009). Universalized narratives: patterns in how faculty define engineering. Journal of Engineering Education, 98(4), 309319. Priem, J., & Hemminger, B. (2012). Decoupling the Scholarly Journal. Frontiers in Computational Neuroscience, 6(19). Raymond, E. S. (2012). How to become a hacker. Retrieved from http://www.catb.org/esr/faqs/hackerhowto.html#believe2 Raymond, E. S. (2001). The cathedral & the bazaar: musings on Linux and open source by an accidental revolutionary. Cambridge, Mass.: OReilly. Sarasvathy, S. D. (2001). Causation and Effectuation: Toward a Theoretical Shift from Economic Inevitability to Entreprenurial Contingency. The Academy of Management Review, 26(2), 243263. Scarantino, A. (2003). Affordances explained. Philosophy of Science, 70, 949961. Searls, D. (2009, August 6). Open vs. Fauxpen. Linux Journal. Retrieved from http://www.linuxjournal.com/content/open-vs-fauxpen Singh, J. V. (1986). Performance, Slack, and Risk Taking in Organizational Decision Making. The Academy of Management Journal, 29(3), 562585. Stallman, R. (2012). The Free Software Definition. Retrieved from http://www.gnu.org/philosophy/free-sw.en.html Steele, C. M., & Aronson, J. (1995). Stereotype threat and the intellectual test performance of African Americans. Journal of Personality and Social Psychology, 69(5), 797811. doi:10.1037/0022-3514.69.5.797
Stevens, R., OConnor, K., Garrison, L., Jocuns, A., and Amos, D. (2008). Becoming an Engineer: Toward a Three Dimensional View of Engineering Learning, Journal of Engineering Education, 97(3), 355-368.

Streveler, R. A., & Smith, K. A. (2006). Conducting Rigorous Research in Engineering Education. Journal of Engineering Education, 95(2), 103105. von Hippel, E. A., & Lakhani, K. (2001). How Open Source Software Works: Free User-to-User Assistance? SSRN Electronic Journal. doi:10.2139/ssrn.290305 von Krogh, G., Spaeth, S., & Lakhani, K. R. (2003). Community, joining, and specialization in open source software innovation: a case study. Research Policy, 32(7), 12171241. doi:10.1016/S0048-7333(03)00050-7 Wenger, E., McDermott, R. A., & Snyder, W. (2002). Cultivating communities of practice: a guide to managing

knowledge. Boston, Mass.: Harvard Business School Press. Wenger, E. (1998). Communities of practice: learning, meaning, and identity. Cambridge, U.K.; New York: Cambridge University Press.