The effects of the internet and open access 1

The Effects of the Internet and Open Access on the Usage of Grey Literature: A Review of the Literature Dawn Mackey, Christina Magnifico, Kathryn Tyler LI810XS

The effects of the internet and open access 2 Objectives: The literature review should help information professionals and researchers understand the validity of grey literature; the relationship between grey literature usage and accessibility; and provide some insight into the evaluation of digital grey literature repositories. Method: This literature review will examine the use of grey literature in an increasingly electronic environment, by utilizing citation analyses, webometric and bibliometric studies, and various supplemental articles that show how the usage of grey literature has changed with the evolution of the digital era. Results: Accessibility of grey literature has increased with the advent of open access and the Internet, but its usage has remained relatively flat. The literature shows that it is important for digital repositories to be consistent for academics to see them as a valid resource. The lack of organization throughout grey literature collections creates problems within the academic community, due to collocation issues. Key Concepts: evaluation of grey literature, digital repositories, printed vs. digital ephemera, open access/free access Variables: perceived value, awareness, user communities

Abstract This paper looks at the validity, usability, and reliability of grey literature in light of increased access due to digital technologies. Many researchers have evaluated usage of grey literature using webometrics, citation analysis, and bibliometrics. This paper also briefly defines grey literature and provides examples of types of grey literature archived within the academic community. The studies in this paper show that the usage of grey literature in academic publishing has remained stagnant even though access to it has increased. It also discusses the perceived validity of grey literature within the academic community and the reliability of grey literature repositories.

The effects of the internet and open access 3 The Effects of the Internet and Open Access on the Usage of Grey Literature This papers research question examines how the advent of the Internet and new technologies has allowed for increased access to previously hard-to-find grey literature. The number of unpublished conference proceedings, dissertations, newsletters, and technical reports has become increasingly accessible to researchers, in both developed and undeveloped countries. Many researchers have seized the opportunity to disseminate their work to the masses via electronic means. However, the increase of grey literature has brought a complex question to the forefront: does the academic community view grey literature as valid, has the usage of grey literature increased in developed and undeveloped countries due to access, and are digital grey literature repositories seen as a reliable source of information retrieval? This literature review will examine the use of grey literature in an increasingly electronic environment, by utilizing citation analyses, webometric and bibliometric studies, that show how the usage of grey literature has changed with the evolution of the digital era. The literature review shows that validity perceptions and use vary by academic field; however, repositories are taking measures to increase validity. Though observations of usage vary by both researcher, and whether it takes place in developed or undeveloped nations, bibliometric studies show that access to grey literature is important regardless of geographic location. Finally, this review found that consistency in collection and distribution of data is what most affects repositories. Research Methodology The research methodology used in this literature review used the terms: gr*y literature AND Internet within the LIS and all EBSCOhost databases available through Emporia State. This query netted 750 results. By applying the Full Text and Scholarly (Peer Reviewed) Journals filters, the authors culled the articles to a much more manageable number of 174.

The effects of the internet and open access 4 In order to focus the articles further, the authors systematically reviewed the abstracts and found 75 articles that were relevant to the research topic. The authors reviewed and uploaded these articles to a bibliographic management tool where, based on their titles, they chose 66 articles. The authors then divided the list of 66 articles among themselves, then read, annotated and contributed comments to a shared database of 30 relevant articles (Figure 1). After they vetted each article for relevancy, the authors skimmed the references of the assigned articles and added the additional sources found (Figure 2). Defining Grey Literature Grey literature is a term used extensively in the world of library and information science. It is defined by the Grey Literature Conference, Luxembourg, as a relatively recent collective noun for information produced on all levels of government, academia, business and industry in electronic and print formats not controlled by commercial publishing i.e. where publishing is not the primary activity of the producing body (Lawrence, 2012, p. 2). Many fields use grey literature within the context of their research. The health sciences, and in particular public health relies significantly upon grey literature (Banks, 2006, p. 7). Other disciplines use grey material as well, including engineering, public policy and political science, and physics. These fields of study rely on freely disseminated works and open access materials to, share key information with relevant parties on what works, in the form of technical specifications, project outcomes, and changes to policy or legislation [quickly and easily] (Lawrence, 2012, p. 2). Grey Literature as a Valid Source of Information Perceptions of the validity of grey literature differ among the various academic fields. De Blaaij (2008) concluded that grey literature is lesser used in the humanities than the hard sciences (p.7). Furthermore, the use of grey literature appears to be much stronger in the

The effects of the internet and open access 5 [social science] oriented history journals (De Blaaij, 2008, p. 6). This finding has less to do with accessibility, and is due in large part to perceptions of scholars in the humanities. De Blaaij (2008) introduced this theory in writing, perhaps it suggests that OA journals are [treated] less serious[ly] by historians. Several studies noticed that OA [j]ournals arent well known in the humanities . . . The printed journals do have a more prestigious image in the humanities (p. 7). Humanities scholars tend to revere printed journals over digitally accessible journals. Many researchers still view open access as a less valid source for publishing their research as, publishing a peer reviewed article in a prestigious journal remains the highest validation for a work of scholarship (Banks, 2006). Other researchers, such as Henderson (2013) noted that, At [this] same time, however, grey literature in the form of research reports, dissertations and other forms of unpublished documents are appearing on the Internet at an exponential rate and garnering considerable attention among researchers (p. 1). As time progresses: Grey literature [is] at the forefront of what [is becoming] the open access movement, forcing many involved in scholarly research to question the dominance of peer-reviewed journals, their value to the public purse, their efficacy for research dissemination and their claims to quality and accuracy. (Lawrence, 2012, p. 126) In order to increase validity and academic oversight within grey literature publishing, many organizations are implementing a process similar to peer reviewing. The Working Group on Expanding Access to Published Research Findings addressed the subject of validity by declaring that what needs to be done is to, put in place arrangements to gather and analyse reliable, highquality and agreed [upon] indicators of key features of the changing research communities

The effects of the internet and open access 6 landscape . . . (p. 8). In stressing the words reliable and high-quality, it highlighted the importance of ensuring that open access materials are valid. Others within the academic community understand that "GL is not subject to the rigors and limitations of commercial publishing, and is therefore more forthright than other sources of information" (Anderson, 2001, p. 82). Thus, it may appear to some researchers that the information contained within a grey literature resource does not align with current research on the particular topic of interest, making it more valuable. One of the reasons why grey literature is seen as valid research, and is often cited in peer-reviewed articles is that, although [it is] not considered a scholarly form of publication, grey literature is often produced by experts in the field. Including grey literature in an article provides a means whereby a researcher can minimize bias in a comprehensive search" (Pappas & Williams, 2011, p. 229). Simply because access to grey literature has increased, that does not necessarily mean that there would no longer be a distinction between it and the peer-reviewed literature" (Banks & Ehrman, 2007, pp. 5-6). Even if the line between grey and white is continually blurred, Schpfel contends that this only applies to potential usage, not to value or quality. He points to a study by Banks (2005) in which he states, even if the hierarchy between grey and white may shift into a continuum of scholarship, this hierarchy will not completely disappear insofar [as the] institution and faculty generally prefer published and peer reviewed documents. As grey literature becomes increasingly accessible, peer-reviewed articles will continue to cite it and show it to be valid research. Usage of grey literature is on the rise, because a vital part of producing worthwhile research is in locating hard-to-find material with the help of Web resources and information technologists. In certain fields such as, but not limited to, fire literature, grey material is becoming a force [to be reckoned with] (Henderson, 2010, p. 4).

The effects of the internet and open access 7 Grey Literature Usage Regardless of accessibility, usage of grey literature varies by field of academic specialty (Di Cesare & Ruggiere, 2006). Some fields of study produce more grey literature than others. For example, medicine and biology create less grey literature than do other fields such as aerospace or agriculture (Pelzer & Wiese, 2003). Therefore, usage varies based on the production of grey literature in any given field of study. Some research argues use of grey literature is increasing, while other research has found that its usage is either unable to be properly measured or is decreasing. Lawrence (2011) wrote about the success of the Australian Policy Online (APO) repository. She outlined the increasing user rate, saying, APOs readership is growing by at least 30% annual and the site now attracts over 65,000 visits each month, and over half a million for the year. The breadth of APOs scope makes it a central resource, and shows it to be a good model to study for increasing user rates. In other resource archives, grey literature downloads have significantly increased (Schpfel & Stock, 2008). A few researchers imply that increased access correlates to higher citation rates; however, researchers have not completely proven this (Pelzer & Wiese, 2003). Other research has found that open access does not translate into higher use. Frandsen (2009) performed a citation analysis and found, The result of this study does not provide evidence of an open access advantage for working papers in economics. Other factors could be causing the increased citation numbers of grey literature. Additionally, citations are not the only measure of usage. Other researchers contend that the demand for grey literature is decreasing even though access is increasing. Even with extensive exposure, grey literature still often languishes unnoticed (Boukacem-Zeghmouri & Schpfel, 2006; Banks, 2006).

The effects of the internet and open access 8 Grey literature is not readily accessible everywhere in the world. Accessibility is the main hurdle to usage in developing nations. For example, availability in Swaziland is modest (Muswazi, 2001). In those instances, hard copy grey literature materials remain necessary. There is not enough worldwide open access due to lack of technological advances and online services in many developing nations (Working Group on Expanding Access to Published Research Findings, 2013). There has been a call to increase open access on an international level. Other barriers to access include a lack of deposit laws in some countries, which prevents the systematic collection and availability of grey literature, and a lack of translation of grey literature in various languages (Anderson, 2001). The Reliability of Digital Grey Literature Repositories Though repositories and online archives are making grey literature more accessible, some researchers question the reliability of these resources. Inconsistent collection and distribution of various libraries and repositories is affecting the usage of grey literature. Lawrence (2011) commented that: Unlike journals and books, the absence of commercial stakes has contributed to a situation where the way in which the different types of grey literature are referenced still depends more on choices made by the bodies that produce, collect, or distribute these documents than on any national or international standard . . . In Australia there is still no systematic collection or preservation of grey documents, or any agreement on how this should be done. (p. 45) Schpfel and Stock (2008) further discuss the problems of grey literature cataloging, stating that "compared to traditional cataloguing standards, metadata for grey material are less

The effects of the internet and open access 9 specific or again, simply missing. This is a problem for referencing, efficient search strategies and evaluation" (p. 26). Lawrence (2012) later surmises that: The advent of the internet has solved some old problems by making it easier to procure many reports, but also created new ones--like keeping track of the increased number of reports. However, the internet has not solved the problem of access to grey literature. (p. 122) This is, in part, due to a lack of organization, distribution and promotion, in both traditional and online access. One of the challenges of the Australian Policy Online (APO), along with any other prominent digital repository, is in the lack of widely agreed-upon standards. Lawrence (2011) suggested a way to increase the reliability of grey literature in repositories by proposing that, perhaps we should be looking at a rating system that might provide options such as: ground breaking, well-written, good synthesis, what works etc. In this way the site might incorporate a web 2.0 type of peer-review for grey literature (p. 48). Increasing reliability is pertinent, because there is uncertainty about what evaluation criteria should be used as there is no clear hierarchy to evaluate grey literature (p. 49). In contrast, Schpfel, Bescond & Prost (2012) expound upon the perceived reliability of deposits into these repositories by mentioning they offer more security and long-term accessibility of unpublished material (p. 11). Pappas and Williams (2011) also consider the reliability of such repositories; when describing several repositories and search engines for grey literature, they talk about such sites being "well-vetted" and "comprehensive". However, the

The effects of the internet and open access 10 sheer number of repositories coming to existence shows that organizations and users alike are seeing some evidence of reliability. Even though repositories still have reliability issues to conquer, their growth is inevitable. Banks and Ehrman (2007) say, "The movement to construct institutional repositories at many universities is a response to the potentially limitless range of digital scholarship" (p. 5). Users of grey literature are already putting this limitless range to the test, as Henderson (2013) points out, Research organizations and universities are capitalizing on this opportunity, actively making their research available on the Internet (p. 1). Henderson (2013) goes on to say that: Over the past decade IRs have grown exponentially and achieved a great degree of success and the reasons for this are obvious. First, in terms of content, the only requirement is a digital version of the publication. Secondly, the Internet provides an unlimited and ever-evolving platform for researchers to build worldwide collaborations in the name of science. Finally, IRs and open-access literature oer scientists unprecedented visibility for their work and, in doing so have changed the nature of scientic communications. (p. 2) Conclusion The results of this literature review illustrate how different academic fields have distinct opinions on the validity of grey literature. Librarians and scholars are working to increase validity by utilizing grey literature within academic, peer-reviewed publishing. Studies that use citation analysis (Muswazi, 2001; Ceasare & Ruggieri, 2006; De Blaaij, 2008; Frandsen, 2009; Henderson, 2013) show a continued use of grey literature within the academic community. Various studies (Pelzer & Wiese, 2003; Boukacem-Zeghmouri & Schpfel, 2006; De Blaaij,

The effects of the internet and open access 11 2008; Lawrence, 2011) concluded with differing results regarding the question of whether or not usage is increasing. Studies (Anderson, 2001; Muswazi, 2001; Schpfel & Stock, 2008) which explored grey literature usage in developed and undeveloped nations find that location also affects grey literature. In undeveloped nations where access is limited, accessibility remains an important issue. Finally, this review found that consistency in collection and distribution of data most affects the perceived reliability of repositories (Banks, 2007; Kroth et al, 2010; Lawrence, 2011; Schpfel et al, 2012; Working Group, 2012; Henderson, 2013).

The effects of the internet and open access 12 References Anderson, B. (2001). Grey literature and electronic publishing. Behavioral & Social Sciences Librarian, 19(2), 8185. doi:10.1300/J103v19n02_07 Banks, M. A. (2006). Towards a continuum of scholarship: The eventual collapse of the distinction between grey and non-grey literature. Publishing Research Quarterly, 22(1), 411. Banks, Marcus A. (2007). How the success of open access publishing can stimulate improved access to grey literature. Urban Library Journal, 14(1), 22. Boukacem-Zeghmouri, C., & Schpfel, J. (2006). Document supply and open access: An international survey on grey literature. Interlending & Document Supply, 34(3), 96104. doi:10.1108/02641610610686012 Ceasare, R. D., & Ruggieri, R. (2006). The evaluation of grey literature using bibliometric indicators: A methodological proposal. In Information for Responsible Fisheries: Libraries as Mediators (pp. 5763). Presented at the 31st Annual Conference of the International Association of Aquatic and Marine Science Libraries and Information Centers, Rome, Italy: CNR Istituto di Ricerche sulla Popolazione e le Politiche Sociali. Retrieved from https://darchive.mblwhoilibrary.org/bitstream/handle/1912/1322/proc06057.pdf?sequence=1 De Blaaij, C. (2008). The use of grey literature in historical journals and historical research: A bibliometric and qualitative approach. Grey Journal (TGJ), 4(1), 1420. Frandsen, T. (2009). The effects of open access on un-published documents: A case study of economics working papers. Author manuscript, Copenhagen, Denmark. Retrieved from http://hprints.org/docs/00/35/23/69/PDF/OA_advantage_for_WP.pdf Gelfand, J. (2007). Grey literature: Taxonomies and structures for collection development. Grey Journal (TGJ), 3(1), 716.

The effects of the internet and open access 13 Henderson, I. (2013). Open-access and institutional repositories in fire literature. Fire Technology, 49(1), 155161. doi:10.1007/s10694-010-0198-1 Kroth, P. J., Phillips, H. E., & Hannigan, G. G. (2010). Institutional repository access patterns of nontraditionally published academic content: What types of content are accessed the most? Journal of Electronic Resources in Medical Libraries, 7(3), 189195. doi:10.1080/15424065.2010.505515 Lawrence, A. (2011). A bridge over grey waters: Collecting and preserving public policy research. Grey Journal (TGJ), 7(1), 4554. Lawrence, A. (2012, May). Electronic documents in a print world: Grey literature and the internet. Media International Australia incorporating Culture and Policy, (143), 122+. Muswazi, P. (2001). Illusion of internet grey literature boon for developing nations: A case study. INSPEL, 35(4), 217225. Myska, M., & Savelka, J. (n.d.). A model framework for publishing grey literature in open access. Pappas, C., & Williams, I. (Jul-Sep2011). Grey literature: Its emerging importance. Journal of Hospital Librarianship, 11(3), 228234. doi:10.1080/15323269.2011.587100 Pelzer, N. L., & Wiese, W. H. (2003). Bibliometric study of grey literature in core veterinary medical journals. Journal of the Medical Library Association, 91(4), 434441. Schpfel, J., Bescond, I. L., & Prost, H. (2012). Open is not enough: A case study on grey literature in an OAI environment. Grey Journal (TGJ), 8(2), 112124. Schpfel, J., & Stock, C. (2008). Grey literature in French digital repositories: A survey. Author manuscript, Charles de Gaulle University of Lille 3. Retrieved from http://hal.archivesouvertes.fr/docs/00/37/92/32/PDF/GL10_Schopfel_Stock_final.pdf

The effects of the internet and open access 14 Working Group on Expanding Access to Published Research Findings. (2012). Accessibility, sustainability, excellence: How to expand access to research publications. Retrieved from http://www.apo.org.au/node/29938

The effects of the internet and open access 15 Appendix

Figure 1. An example of the spreadsheet in which the authors compiled their literature.

Figure 2. An example of the spreadsheet in which the authors vetted their sources.