Routledge cities Communication Studies ISSN: (Print) (Online) Journal homepage: htps/ww.tandifonline.com/lo\rest20 Challenges and Best Practices Associated with Sharing Research Materials and Research Data for Communication Scholars Nicholas David Bowman & Patric R. Spence To cite this article: Nicholas David Bowman & Patric R. Spence (2020) Challenges and Best Practices Associated with Sharing Research Materials and Research Data for Communication. Scholars, Communication Studies, 71:4, 708-7 16, DOI: 10,1080/10510974,2020,1799488 To link to this article: https://doi.org/10.1080/10510974.2020.1799488 era ee CF submit your artice to this journal lal Article views: 1256 FW View retotes ates @ von ronmarsaa EB) cing ances: 8 View cting articles CP Full Terms & Conditions of access and use can be found at https:www.candfonline.com/action/jaurnalinformation?journalCode=rest20,Soon ou 70470876 JQ Routledge pit ens 220179048 yore Challenges and Best Practices Associated with Sharing Research Materials and Research Data for Communication Scholars Nicholas David Bowman ®* and Patric R. Spence @! ‘College of Medla and Communication, Texas Tech Unversity, Lubbock, TX USA; *cholson School of Communication and Media, Unversity of Central Florida - Dovintown, Orlando FL, USA ‘agsTRACT kerwonos Scientific practices are constantly evolving to meet the best practices Open slence pacts: data ‘and standards available at a point in time. In the current publication sharing ty oat ‘environment, advances in communication technology in tandem with Tansrarency scenic ‘concerns about the transparency of extant communication scholarship Paton have encourag: how we share knowledge with ‘one another ‘Communication science, this essay summarizes and explains how feseorchers can critically consider so-called “open materials” and “open data’ when publishing in Communication Studies as well as ‘other communication and adjacent journals. Scientific inquiry is an evolving process. As new methods and techniques are learned for both (a) making empirical observations about the world and (b) analyzing those observa- tions, we improve and refine our practices so that we can have confidence in our efforts to describe, explain, predict, and even control the phenomena around us. As a byproduct of cour day-to-day engagement with science (Kuhn, 1962, refers to this as normal science), we are bound to uncover novel, curious, or concerning deviations from what we presume to be “truth” (for example, data that rejects a given hypothesis or that cannot be easly explained by established theory: Popper, 1959). For Kuhn (1962), these deviations are what eventually ‘compel us toward broad reconsiderations of science ~ scientific revolutions being the result of radical rather than gradual changes to (accepted as) normal science, Although Kuhn was referring specifically to theoretical paradigms, parallels can be drawn for how we under- stand scientific practices as well. For example, Greenwald (2012) observed that the majority Cf Nobel science awards in physics, chemistry, and medicine were awarded for methodo- logical rather than theoretical contributions, suggesting that the robustness of our theories rests on the veracity of the methods used to generate and test them. ‘of open science practices wwe been pence, 2019) By transparency, we refer to purposeful ‘CONTACT PaticR. Spence @ spencesuctedu Nek Bowman @ ickbownanattuedycoMMUNIcATION STUDIES @) 709 Lewis (2019) and Deinlein etal. (2020) engage the Center for Open Science's nomenclature of “open materials” and “open data” when referring to these respective practices. Bowman and Keene (2018) suggested an ordinal-level model of transparency in which the du jour standard claim that study content is available “by request to Author” is relatively opaque, sharing study materials is more transparent, and sharing study materials and data are increasingly more transparent." In this editorial, we summarize what we mean by shared materials and shared data and discuss the costs and benefits associated with both practices” In this essay, we also recognize the remarkable epistemological and methodological diversity within communication studies that will not always allow for implementing all, some, or any of the suggestions offered herein. However, we do hope that this editorial helps («) further define open materials and open data practices, (b) offer some conerete ways in which researchers can carefully consider and translate these concepts into their own research practice, and (¢) encourage more critical and creative thinking about how to amend, extend, or develo e in research. Most importantly, we se .e Bowman & Keene, 2018). Sharing Materials In simplest terms, open materials refers to practices in which authors publish some or all of their research materials in a manner that makes them available freely available and without qualifications to interested readers. One way to think about open materials is to consider the Method section typical to ‘manuscripts published in accordance with the American Psychological Association style ‘guide (the standard formatting for empirical esearch on human communication). In the ‘Method section, researchers provide details of their study participants and sampling frames (including recruitment procedures), procedures, stimulus materials (if any were used), and _measures of key variables pertinent to the work. A study method section should be detailed enough so that other scholars could reconstruct the work for themselves, either in « conceptual or literal sense (discussed at the conclusion of this section). Of course, there are inherent limitations to the fidelity at which we can descript our study methods using text alone. Likewise, journals (including digital publications) have space constraints that require authors to truncate the more granular details of their study methods. Some of the more ‘obvious casualties include the text for scale items ~ most manuscripts report a few sample items and refer readers to another manuscript for reference, and often times those refer- enced manuscripts only contain items “by request to Author.”* Details such as audio-visual or interactive stimuli can be cumbersome to explain in text, and yet a reader’s ability to understand and accept prima facia the validity of a given manipulation hinges on their ability to understand the manipulation itself, Moreover, some study documents might not seem critical to how we understand a given project, but can be illuminating nonetheless ~ documents such as confederate scripts, reproductions of survey documents as they710 © N.0.80WMAN AND P. SPENCE appeared to participants, sketches of laboratory designs, content analysis codebooks and ‘more (depending on the study design, naturally). The ability to recreate much of the currently published research in communication studies based on summarized paragraphs from methods sections in published articles would be difficult, However, if authors were to publish this content in a supplemental resource folder along with their published manuscript, research transparency is greatly increased. Many colleges and universities house research repositories for providing secure and persistent web addresses for these materials, and a growing number of third parties provide similar services ~ many, such as the Github (github.com) and Open Science Framework (osfio) do so at no cost to the researcher.* Some of these services even allow ‘materials to be published with a Digital Object Identifier (DOI) ensuring that the authors can be properly cited. Somewhat ironically, the use of supplemental shared materials spaces can allow authors to be brief with the content of their methods sections so that they can devote more of their manuscripts toward opening and closing arguments. Especially given the increased reliance on digital documents such as.pafs and even fully online journals, immediately linking a manuscript to its online supplemental files is increasingly seamless ‘An additional benefit of sharing research materials is that it facilitates the replication of published research, Replication efforts are critical for understanding the validity of the current scientific record, as they tell us about the stability of our accumulated knowledge over time (McEwan et al, 2018). Whether scholars are interested in exact replications (in which an individual study protocol is replicated with complete fidelity, usually a test of| stability) or conceptual replications (in which a given study protocol is replicated by some aspect of sampling frame, stimulus, or measurement is altered, usually a text of boundary conditions), we are reminded of the fact that no knowledge is absolute. Replication is among the strongest tools we have to continually recalibrate the scientific record, as it is heavily tied to the process of falsification (Popper, 1959). As reported by Keating and ‘Totzkay (2019), replication efforts in communication are uncommon, and in one of the few focused publications on replication (a special issue of Communication Studies; McEwan et al, 2018), only two of nine studies were able to replicate the initially published results. Here, we note that a lack of replication does not automatically suggest anything untoward about prior research but does suggest that knowledge of a given phenomenon needs updating, Likewise, replication efforts are greatly facilitated when researchers make openly available their study materials. Open Data In simplest terms, data sharing is the practice of making empirical data from research studies frely available and without qualifications. As with shared materials, shared data can >be housed in supplemental online folders alongside a published manuscript either as part of| ‘that manuscript or under a separate DOI. When data is shared, as one is reading the results section of a given manuscript, they have on-demand access to the data files from which those reported analyses are derived. Manuscripts are limited in terms of how much detail an author can delve into with respect to describing their data. As a result, authors will report only what they feel that readers would either (a) want to know or (b) need to knot n this results in reports of only summary findings: basic sample descriptive, nomcoumunicarion stuns @) 71 information for statistical significance tests (such as truncated or rounded p-values), or the ‘use of exemplars to represent trends in open-ended data ‘A cursory review of many publications would suggest that there isa general commonality among researchers in terms of many statistical results that are reported. However, we can identify a few concerns, Broadly, authors and reviewers commonly disagree on which data is or is not important for a study ~ speaking to quantitative work for a moment: some disagree on how p-values should be reported and others disagree on which effect sizes should be reported and stil others debate on whether or not bivariate correlation tables should be reported in all manuscripts (Benjamin et al, 2017; Lakens, 2013: Sawilowsky, 2009). Providing datasets shortcuts some of these concerns insofar as curious readers can calculate the (for them) missing data analyses for themselves, Extending a bit further, data sharing aids greatly toward ensuring the reproducibility of published results ~ that is, the ability for readers to use shared datasets to obtain the same results as those published (Asendorpf et al, 2013). Best Practices with Sharing Materials and Data Having discussed the basics of open materials and open data, we offer a few suggestions below related to how sharing can be done to increase transparency, facilitate replication and reproduction, and to protect human participants in our research. We also add a note on sharing to facilitate the organization of research efforts, internally and externally. Sharing Easy-to-understand Content In many respects, best practices for sharing materials and sharing data are similar. As the audience for your materials is essentially “anyone interested,” using clear and unambiguous language and labels forall ile and folders is important. Specific to research data, Wickham (2014) define tidy datasets as those that clearly link the structure of a dataset with its meaning: for example, each row of data might representa single study participant, and each ‘column might represent a unique (and clearly labeled) variable. Although some advocate for standardized practices that would be common to all data, we merely suggest that researchers make clear whichever structure and labeling they are using - creating and sharing data codebooks (such as annotating survey documents or content analysis guides) can even further facilitate a reader's understanding of a dataset. For sharing materials and data, one suggestion would be to provide them to a colleague before posting to a repository and determine if they are easy to understand. Shared materials should be understandable such that replication ofthe work (ether asa mental exercise or as a planned research practice) is possible using only the content at hand and likewise, shared data should be understandable stich that any reader could reproduce the results of the author's manuscript without additional clarification. Sharing for Compatibility Most of us work with proprietary software as part of our normal research practice ~ Microsoft Office suite for manuscripts and spreadsheets, Adobe Creative Cloud for image and video manipulation, Qualtries for online surveys, or one of a number of different2a © wo.sowmanano a sPence statistical packages (such as SPSS or SAS) for data cleaning, analysis, and reporting, Thus, sharing materials or data from these programs can hinder access for others who don't have the programs. Whenever possible, we advocate for sharing content using various file formats to ensure compatibility across a spectrum of users and software packages. Some files types are more standardized than others, such as docs (which can usually be opened by various programs), pdfs for formatted content, .mp3 for audio files, and .csvor .tsv (comma separated values and tab separate values, respectively) for data files. Providing files in multiple formats can also be quite useful, for example, SPSS-based analyses might include both the sps dataset (as wel as av or v version) and the spy analysis output file (as well as.a pdf or doc version); if this analysis was based on a Qualtrics-hosted survey, then the asf (Qualtrics-compatible survey file) could also be shared (as well asa .pdf or «loc of the survey itself). Sharing for Confidentiality (such as date of birth, geographic location, marital status, and gender), Indeed, most institutional review boards Issues of participant confidentiality were at the forefront of 22016 event in which a researcher uploaded the personal profiles of nearly 70,000 OkCupid users (a popular online dating service) to the ostio repository (Resnick, 2016). Here, we offer few simple considerations for shared data In such cases, one possibility is to that share the same statistic of original data (see Quintana, 2019), «As with above, ll and unadulterated data files) M0) identifier D (Wiekham, 2014). asets used for analyses, although we would strongly encourage a detailed accounting of the steps taken to “clean” the dataset. Many institutional review boards will offer to review a dataset prior to sharing publication, to check for potential identifiers and other concems related to human participants protection. though ‘concerns over confidentiality and data security (as well as participants common insteniveness gates with he conse procu:). Ruiagamaaagay with participant ,e st a (eg. @ sample file prior to the study starting or even their actual data). Flory and Emanuel (2004) found evidence that direct conversations over consent can increase participant understanding of the research process, but this method is likely restricted to studies in which there are direct participant-researcher interactions. Broadly speaking, we must ensure thatcommunicarion stunts @) 712 Serene 1ot) to participate in the research. eer eee i oe i aa i ee ee yi Se ee ea ear ates es ean en populations (Gelhert & Mozersky, 2018; Quinn, 2015) - an indeed, doubts about So and ines scae acme a aoa ata ola research data in all cases (regardless of the nature of the data collected, such as data fg eee gees cuca anes nar Sint mcs niet conuir sharing nmr antsy sch as ht tons Gy sage mcg a a ene seg ee ca am controversies about when and how to engage participant consent - for example, ae coon ear ane coer aia chalingog. gies tek ndielcharognpematonin ness ( public forum) might not be aware that this information could be co-opted for another ee ea fea none ee ero ‘ton coulda pony tel to ak va resialction Tes i Maeeaoa ciao tere eas apenas amas Gh erarecoarematiraan ang (eumire ata) Sharing for Organization [As overheard during a recent panel on open science practices, “your worst co-author is Yourself, six months ago” (Fisher, personal communication, May 20, 2020). To this end, fone ancillary benefit of sharing materials and data is that they help research teams track their own projects on a readily accessible server. Related to this, research can also demon- strate the journey of their published research, for example, sharing earlier drafts of many scripts that might have been presented at conferences or professional associations. One unique benefit of using sharing service to organize research can be found for researchers who conduct several analyses from single data collections, There are numerous reasons for this approach, as we cannot assume a 1:1 relationship between the efforts that go into a specific data collection and the scholarship that these efforts produce. In an effort to avoid allegations of “salami slicing” in which multiple manuscripts are published from shared datasets but those manuscripts make less-than-nominal unigue contributions to science (Collyer, 2019), one suggestion is to make the dataset publicly avaiable as a citable document (Ge, with a DOD); this would be similar for research that reports on secondary data analysis, Directing multiple supplemental projects to each other can also make it clearer how manu- scripts that share a common dataset are making unique contributions to communication studies. For researchers concemed about having exclusive rights tothe initial manuscripts that emerge from their datasets, one possibilty that balances transparency with exclusivity is to release portions of the data analysis file for so-called marker papers (the first papers out of 4 dataset) and then embargoing the fll dataset fora period of time — such is a controversial- discussed elsewhere that balances known inequities in career incentives with broader calls to improve science (Portugal & Pierce, 2014). Another concern is that providing publicly available data may impact the anonymous review process. On the one hand,74 © W.0.80wmAN AND PR SPENCE most datasets do not include information that personally identifies the research team and likewise (or at leat, identifiers that could not be removed), and most sharing services (such as ‘OSF and Github) have settings so that researchers can share their materials anonymously for peer review. During the peer review process, anonymized datasets would be subjected to the same confidentiality agreements as the manuscripts being reviewed and thus, reviewers should not be able to make use of them for any reasons beyond peer review. On the other hand, because sharing materials and data is not a requirement for submission to many journals, author(s) can also inform the eltor within the cover letter that if published, the author(s) wish to make their materials and data available. Then the specific points of this can be worked out luring the production process. However, reviewers might object to elements of a manuscript being published that were not also part of the peer review process and could claim that keeping materials inaccessible during peer review diminishes the quality of peer review Conclusion ‘The Greek philosopher Heraclitus reminded us that the only constant in life was change. Indeed, this maxim holds true for the production and distribution of science. Changes are inherent to the creation and certification of knowledge, asthe veracity of our claims about communication are only as strong as the research that buttresses them. Just as commu- nication has emerged from radical advances in theory and method, communication scholar- ship is well-positioned to leverage the transparency of open science practices to better establish the contributions of our scholarship. Simply stated, we encourage Communication Studies authors to publish more of their research ~ not simply more manuscripts, but more of the “stuff” that those manuscripts are based on, The open sharing of research material and research data help move sciences toward transparency, which helps us better under- stand our collective and iterative knowledge. ir model, the most transparent form of academic publication involves the pre-registration ‘of study hypotheses, design, and data analysis plans. Lewis (2019) and Dieolin etal (2020) discuss pre-registration more extensively, and the Center for Open Science offers expanded tails on the process of pre-registration at https/wwww.cos.o/prereg. Notably, this manuscript is intended as a quick primer on data sharing practices rather than ‘a comprehensive review of those practices. We refer readers to Deinlein etal. (2020) for a more detailed discussion of an agenda for open science practices in communication research, 3. This can be even more complicated when a manuscript amends or edits items, such that referencing the original item set is potentially moot. 4. Both Klein et al (2018) Deinlein etal. (2020) offer numerous examples ofthese services and others, Disclosure Statement [No potential conflict of interest was reported by the author(s)coumunicerion suns @) 715 Notes on contributors Nicholas David Bowman (Ph.D., Michigan State University) isan Associate Professor inthe College ‘of Media and Communication at Texas Tech University. His research is primaeily focused on the cognitive, emotional, physical, and social demands of interactive media such as video games and virtual reality. He has co-authored over 80 manuscripts and two textbooks, and recently completed a J. William Fulbright teaching and esearch grant in Taipei, Taiwan. He was most recently the editor of ‘Communication Research Reports and is the incoming editor of Journal of Media Psychology. Patric R. Spence (Ph.D, Wayne State University) is a Professor in the Nicholson School of Communicstion and Media at the University of Central Florids. He is the current editor of ‘Communication Studies and affliated with the Communication and Social Robotics Labs (www combotlabs.org). ‘ORCID [Nicholas David Bowman @ http//orcidorg/0000-0001-3594.9713 Patric R. Spence ( http/Jorcid.org/0000-0002-1793-6871 References Asendorpf, J. B.. Conner, M., De Fruyt, F, De Houwer, J, Denissen, J. J. A. Fiedler, K. ‘Wicherts, J, M. (2013). Recommendations for increasing replicability in psychology. Europea Journal of Personality, 272), 108-119. hitps:|/doi.org/10.1002/per.1919 Benjamin, D. |, Berger, J ©., & Johnson, V. E.. (2017). Redefine statistical significance. Nature ‘Human Behaivor, 22), 6-10. btip:idoi.org/10-1038/s41562-017-0189-2 Bowman, N.D., & Keene, RJ. 2018). A layered framework for considering open science practices Communication Research Reports, 35(4), 363-372. bltpsi//doi.org/10.1080/08824096,2018, 1513273 Collyer, T. A. (2019) “Salami slicing” helps careers but harms science, Nature Human Behavior, 3 (10), 1005-1006 hteps/doi.org/10.1038/s41562-019-0687-2 ‘Cummings, A. Zagrodney, J. M., & Eugene Day, T. (2015). Impact of open data polices on consent ‘to participate in bumian subjects research: Discrepancies between participant action and reported concerns. PLoS One, 10(3) https://doi.org/10.137 /journal pone.0125208 Deinlein, T.. Johannes, N., Bowman, N.D., Masur, P. Engesser, S, Kiimpel, AS... de Vreese, C. (2020), An agenda for open science in communication Fournal of Communication. https//do.org/ 10.1093 jocrqa052/3803422 Flory, J. & Fmanuel, E. (2004) Interventions to improve research participants’ understanding in informed consent for esearch. Journal of the American Medical Association, 292(13), 1593-1601, ups:/doi.org/10.1001/jama.292.13.1593 Franzke, A. S. Bechmann, A, Zimmer, M, & Ess, C, and the Association of Internet Researchers, (2020). Internet research: Ethical guidelines 3.0, Assocation of Internet Researchers. Gelhert, S, & Mozersky, J. (2018). Seeing beyond the margins: Challenges to informed inclusion of ‘vulnerable populations in research. The Journal of Law, Medicine & Ethics, 46(1), 30-43. https:// doi org/10.1177/1073110818766006 Greenwald, A. G. (2012), There is nothing so theoretical as a good method. Perspectives on Psychological Science, 72), 99-109, https:/doi.org/10.1177/174569161 1434210 Keating, D. M., & Totzkay, D. (2019). We do publish (conceptual) replications (sometimes): Publication trends in communication science, 2007-2016. Annals of the International Communication Association, 433), 225-239. https:/doi.org/10.1080)/23808985.2019.1632218 Klein, 0., Hardwicke, T. F., Aust, F, Breuer, |, Danielsson, H., Mohr, AH... Frank, M.C. (2018) ‘A practical guide fr transparency in psychological science, Collabra: Psychology, 4(20)- https doi. ‘org/10.1525/collabra.158716 © Wo.sownan ano. SPENCE Kuhn, 7. (1962). The structure of scientific revolutions. University of Chicago Press. akens, D. (2013). Calculating and reporting effect sizes to facilitate cumulative science: A practice primer for ttests and ANOVAs, Frontiers in Psychology, 26. hitps//doi org/10.3389/fpsyg.2013. 0863 Lewis, N. A. (2019), Open communication science. A primer on why and some recommenda how. Communication Methods and Measures(2), 71-82. htips//doi.org/10.1080/1931248 1685660 ‘McEwan, B. Carpenter, C. J, & Westerman, D. (2018). On replication in communication science Communication Studies, 69(3), 235-241. https/doi.ory/10-1080/10510974,2018, 1464938 Popper. K. (1959). The logic of scientific discovery. Routledge. Portugal, S.J, & Pierce, S. E. (2014). Who's looking at your data? Science, hitps//doi.org/10.1126/ science caredital 400052 ‘Quinn, C. R. 2015). General considerations for research with vulnerable populations: Ten lessons for success Health & Justice, (1). https:/doi.ong/10.1186/s40352-014-0013-2 Quintana, D. S. 2019). A synthetic dataset primer for the biobehavioural sciences to promote ‘reproducibility and hypothesis-generation.elJFE.https//doi.org/10.755d/eLife 83275 Resnick, B. (2016, May 12). Researchers just released profile data on 70,000 OkCupid users without ‘permission, Vor. https:l/web archive.org/web 2016051 6055907 /https//www.vox.com/2016/5/12/ 116661 16/70000-okcupid-users-data-release Rocher, L., Hendricks, . M, & de Montjoye, Y.~A. (2018). Estimating the success of re-identifications in incomplete datasets using generative models. Nature Communications, 10(1), 3069. bttpsidoi, ‘org/10,1038/s41467-019-10983-3 Savsilowsky, S. S. (2009). New effect size rules of thumb. Journal of Modern Applied Statistical ‘Methods, 8(2), 597-599. https doi.ong/10.22237(jmasmn/1257035100 Spence, P. R. (2019). Policy, practices and communication studies. The More Things Change « Communication Studies, 70(1), 129-131. htps!/doi.org/10-1080/10510974.2019.1570715 ‘Wickham, H. (2014). Tidy data. The Journal of Statistical Software, (10). https/doi.org/10.18637/ js.059.10 ions for 2018,