Designing and Testing News Literacy Messages For Social Media PDF

Mass Communication and Society
ISSN: 1520-5436 (Print) 1532-7825 (Online) Journal homepage: https://www.tandfonline.com/loi/hmcs20
Designing and Testing News Literacy Messages for

Social Media
Melissa Tully, Emily K. Vraga & Leticia Bode
To cite this article: Melissa Tully, Emily K. Vraga & Leticia Bode (2019): Designing and
Testing News Literacy Messages for Social Media, Mass Communication and Society, DOI:
10.1080/15205436.2019.1604970
To link to this article: https://doi.org/10.1080/15205436.2019.1604970
Accepted author version posted online: 11

Apr 2019.
Submit your article to this journal
Article views: 3
View Crossmark data
Full Terms & Conditions of access and use can be found at

https://www.tandfonline.com/action/journalInformation?journalCode=hmcs20
Publisher: Taylor & Francis & Mass Communication & Society Division of the Association for
Education in Journalism and Mass Communication
Journal: Mass Communication and Society
DOI: 10.1080/15205436.2019.1604970
t
ip
Designing and Testing News Literacy Messages for Social Media
cr
Melissa Tully*
us
University of Iowa
Emily K. Vraga
an
George Mason University
Leticia Bode
Georgetown University
M
*Corresponding author
ed
Abstract: As concerns grow about the spread of misinformation through social media, scholars
have called for improving the public’s media literacy as a potential solution. This study examines
pt
the effectiveness of deploying news literacy (NL) messages on social media by testing whether
ce
NL tweets are able to affect perceptions of information credibility and NL beliefs. Using two
experiments, this study tests NL tweets designed to (1) mitigate the impact of exposure to
Ac
misinformation about two health issues (genetically modified foods and the flu vaccine); and (2)
boost people’s perceptions of their own media literacy and media literacy’s value to society
broadly. Findings suggest that NL messages are able to alter misinformation perceptions and NL
beliefs, but not with a single message, suggesting the need to develop tailored and targeted NL
campaigns that feature multiple messages and calls to action.

News Literacy Messages for Social Media 2
Keywords: News literacy, misinformation, credibility, self-perceived media literacy, Twitter
Designing and Testing News Literacy Messages for Social Media
Following the 2016 U.S. presidential election, concerns about fake news and
t
ip
misinformation have proliferated, with some suggesting that media and democracy are under
attack (Holan, 2016; Levine & Kawashima-Ginsberg, 2017). The idea that we might have the
cr
power to combat misinformation by improving our media literacy – “our ability to access,
us
analyze, evaluate, create and participate with messages in a variety of forms” (Center for Media
Literacy) – resonated with publics, educators, and researchers looking to respond to a potential
an
“post-truth” crisis (Lewandowsky, Ecker, & Cook, 2017; Mihailidis & Viotty, 2017).
Since then, a number of studies and reports have suggested that increased and improved
M
media literacy is part of a potential solution to combating the spread of misinformation (Bulger
& Davison, 2018; Lewandowsky et al., 2017). Although media literacy is not a panacea, it could
ed
be an effective part of a solution that also includes human and algorithmic fact-checking,
changes in how news is vetted and circulated on social media sites, and government oversight
pt
and regulations, to name a few (Marwick, 2018). In contrast, other critics have gone so far as to
ce
suggest that media literacy education might, in fact, have backfired in part by giving us false
confidence in our abilities to discern fact from fiction (boyd, 2017). Despite the limitations of
Ac
media literacy education (boyd, 2018), research has shown that media literacy curricula and
efforts targeted to audiences outside of classroom settings have been effective at improving
media literacy skills and knowledge, contributing to more thoughtful news consumption
(Fleming, 2013; Klurfeld & Schneider, 2014; Vraga & Tully, 2015).
Much of the recent fodder has focused broadly on “media literacy” with little agreement
on definitions and applications. Far less discussion has centered around specific and tailored
efforts that promote news literacy (NL), a component of media and civic literacy that focuses on
news production, context, consumption, and the role of the press in a democratic society (Craft,
Ashley, & Maksl, 2017; Kovach & Rosenstiel, 2011; Levine & Kawashima-Ginsberg, 2017;
t
ip
Vraga & Tully, 2015, 2016). NL education has been shown to bolster skepticism toward false
and misleading information making it particularly suitable to addressing the spread of
cr
misinformation (Craft et al., 2017; Kahne & Bowyer, 2017).
us
Likewise, much of this discussion has focused on how to improve media literacy as part
of classroom experiences, which divorces such efforts from ongoing attempts to address
an
misinformation circulating online and leaves out the majority of the population not enrolled in
school (Bulger & Davison, 2018). NL interventions, such as digital advertisements and
M
interactive quizzes, need to address the fact that news is often consumed online, increasingly in
social media environments by adults, a context and audience not typically addressed in media
ed
literacy education (Bulger & Davison, 2018; Vraga & Tully, 2016).
Therefore, building on previous research that suggests that NL messages can reinforce
pt
NL attitudes and beliefs and serve as reminders to apply NL skills to news consumption (Tully &
ce
Vraga, 2017; Vraga & Tully, 2015), this study tests the effectiveness of sharing NL tweets that
encourage critical news consumption to mitigate the impact of exposure to misinformation, as

Ac
well as to boost people’s perceptions of their own media literacy and its value in a democratic
society. In other words, can NL tweets affect perceptions of misinformation credibility and news
literacy?
This study uses two experiments to test the effectiveness of NL tweets across different
information contexts, providing insight into both the promise and limitations of NL messages on
Twitter, a platform where misinformation circulates and spreads (Hindman & Barash, 2018;
Marwick & Lewis, 2017). Understanding whether NL messages can influence credibility
perceptions of misinformation has theoretical and practical value for research and practice.
Misinformation and Credibility
Misinformation is commonly defined as “objectively incorrect information” (Bode &
t
ip
Vraga, 2015, p. 621; see also Nyhan & Reifler, 2010). Closely related to this are misperceptions,
defined as “cases in which people’s beliefs about factual matters are not supported by clear
cr
evidence and expert opinion” (Nyhan & Reifler, 2010, p. 305). In other words, misperceptions
us
are beliefs in misinformation. The spread of misinformation and misperceptions is not a new
problem, even if it has gained new salience and significance in recent years (Craft et al., 2017;
an
Hofstadter, 1965; Nyhan & Reifler, 2010). However, concerns have grown with the existence
and growth of social media as a medium of (mis)information dissemination. Due to the speed
M
with which information spreads on social media, misinformation can spread quickly, and outpace
attempts to correct it (Marwick & Lewis, 2017), even if research shows that correction efforts are
ed
often successful (Bode & Vraga, 2015; Vraga & Bode, 2018).
Previous research suggests that experts are more likely than non-experts to successfully
pt
correct misinformation, in part due to their higher perceived credibility (Lewandowsky et al.,
ce
2012; Nyhan & Reifler, 2010; Vraga & Bode, 2018). Thus, changing perceptions of credibility –
in particular boosting credibility perceptions of accurate information and decreasing credibility

Ac
perceptions of misinformation – might be a mechanism by which rejecting misinformation
operates. In this study, rather than focus on correcting misinformation on social media, we
evaluate the promotion of critical evaluation of information as a mechanism for assessing
information quality and affecting credibility perceptions.

NL efforts seek to change perceptions of information credibility – or people’s judgments
of whether information is fair, accurate, unbiased, and trustworthy (Fico, Richardson, &
Edwards, 2004; Meyer, 1988) – ideally bolstering credibility perceptions of high-quality
information and decreasing perceptions of low-quality or false information. The question of
information quality and credibility is particularly critical online, where people often make
t
ip
superficial judgments of credibility based on contextual cues (Bhandari, 2018; Metzger,
Flanagin, & Medders, 2010) and where sources with different credibility appear on the same
cr
screen and compete for attention (Mihailidis & Viotty, 2017; Thorson, Vraga, & Ekdale, 2010).
us
News Literacy
In this study, we focus on news literacy because it emphasizes building the skills
an
necessary to become more mindful news consumers who understand contemporary news
production and consumption (Ashley, Maksl, & Craft, 2017; Kovach & Rosenstiel, 2011; Vraga
M
& Tully, 2016). To become news literate requires an understanding of content, production,
consumption, and contexts, and the ways in which personal beliefs influence interpretation of
ed
news (Craft, Ashley, & Maksl, 2016; Klurfeld & Schneider, 2014).
Research suggests that despite familiarity with navigating online spaces, young people,
pt
including students, struggle to evaluate information online (McGrew, Breakstone, Ortega, Smith,
ce
& Wineburg, 2018), suggesting competency in this area is lacking (boyd, 2017, 2018). Head and
colleagues (2018) found that students get their news from a variety of sources and engage with
Ac
news across a variety of platforms creating diverse news diets that include both traditional and
newer kinds of news. They also found that students know that critically engaging with news
takes effort and that some are willing to put in the time, while others are not (Head, Wihbey,
Metaxas, MacMillan, & Cohen, 2018), a finding that echoes other NL research with both teens
and adults (Craft et al., 2016; Tully, Vraga, & Smithson, 2018). These studies find that although
people have basic NL knowledge and skills (Craft et al., 2016), they fail to apply them when
analyzing news stories (Tully et al., 2018).
Research suggests that exposure to NL messages should affect perceptions of information
credibility (Clayton et al., 2019; Tully & Vraga, 2017; Vraga & Tully, 2015). We expect that
exposure to NL messages should affect perceptions of information credibility in two ways
t
ip
(Kahne & Bowyer, 2017). First, NL messages should increase credibility perceptions of high-
quality information. Reminding people of the markers of ‘good’ and ‘bad’ information should
cr
help them identify each, improving their perceptions of credible information through an implicit
us
comparison to less credible information. Second, NL messages should decrease perceptions of
low-quality information (in our study, the misinformation tweet). NL messages that remind
an
people of the existence and problematic nature of untrustworthy information should help them
recognize poor information and acknowledge it as less credible.

M
These expectations are derived from previous research, which has found that NL
messages can prompt change in perceptions of information credibility for unbiased news but may
ed
exacerbate polarization in credibility ratings of biased news (Vraga & Tully, 2015). For example,
short NL videos, similar to pre-roll advertisements found on video sharing sites like YouTube,
pt
have been effective at conveying NL concepts to diverse audiences, reducing hostile media
ce
perceptions of unbiased news content, and improving perceptions of news credibility (Vraga &
Tully, 2015, 2016). In addition, in their study of the politicization of science, Bolsen and
Ac
Druckman (2015) found that “warning” messages, which “alert individuals about the content of
an upcoming message,” were an effective method to counteract messages that undermined and
challenged scientific consensus (p. 748). Furthermore, research suggests that exposure to NL
education and the knowledge it produces may generate skepticism towards misinformation and
political conspiracy theories (Craft et al., 2017; Kahne & Bowyer, 2017).
A nascent body of research has begun to examine the potential for NL messages to
increase skepticism of misinformation in the context of social media. Kahne and Bowyer (2017)
found that people with greater media literacy were more likely to rate evidence-based posts as
more accurate than misinformation posts, even when both posts matched their political beliefs.
Clayton and colleagues (2019) found that warning people about potentially misleading articles
t
ip
on Facebook affected accuracy perceptions of both misleading and true headlines, suggesting the
need for continued efforts to develop effective messages.
cr
In addition, exposure to NL messages should affect individuals’ NL beliefs (Tully &
us
Vraga, 2017, 2018). Research suggests that exposure to NL messages can affect beliefs about the
importance of media literacy in society broadly and an individual’s self-perceived media literacy
an
(Tully & Vraga, 2018). First, NL efforts are valuable to democratic society if they can encourage
informed engagement with news without leading to cynicism (Kovach & Rosenstiel, 2011;
M
Mihailidis & Thevenin, 2013). Value of media literacy (VML) is an empirical measure that
gauges the value that people place on media literacy in a democratic society (Vraga, Tully,
ed
Kotcher, Smithson, & Broeckelman-Post, 2015). If individuals value media literacy as a social
good, it may promote more critical news consumption as people think about the role of news in a
pt
larger context.
ce
Second, NL efforts and interventions should also encourage people to see themselves as
media literate – able to understand and apply media literacy knowledge and skills in their daily
Ac
lives. Examining self-perceived media literacy (SPML) provides insight into individuals’ beliefs
about their media literacy abilities, which are separate from their actual literacy, knowledge, or
behaviors (Vraga et al., 2015). Research suggests that NL efforts, including exposure to NL
messages, can improve confidence in one’s own media literacy (Chu & Lee, 2014; Tully &
Vraga, 2018), which may be a precursor for applying NL skills to actual news consumption. In
other words, if people feel that they are news literate, this self-efficacy may promote more
critical news behaviors. However, this confidence may backfire when it does not align with
skills, application of those skills, or when it leads people to think they do not need to put effort
toward critically evaluating news and information (boyd, 2017, 2018; Tully & Vraga, 2018).
Ideally NL messages circulating on social media will function as reminders for people to
t
ip
critically evaluate content as they peruse posts on social media sites (Burgoon & Miller, 1990;
Pfau, Kenski, Nitz, & Soreson, 1990; Vraga & Tully, 2015). These messages should remind
cr
people that critically evaluating news and information online is important and that it is necessary
us
to be an active consumer. Research on “reminders” or “warnings” suggests that these messages
can be effective persuasive tools, consistent with research on inoculation and reinforcement
an
messages (Bolsen & Druckman, 2015; Burgoon & Miller, 1990; Vraga & Tully, 2015).
Importantly, these messages represent a proactive step at addressing misinformation and have the
M
potential to influence subsequent processing of varied news and information.
NL messages designed to be consumed as part of regular news consumption on social

ed
media cannot cover all aspects of news literacy so they must focus on key actionable elements
and be designed to resonate with audiences on different platforms. With this in mind, the tweets
pt
in Study 1 focus on audiences’ critical news consumption and evaluation by tapping into the idea
ce
that our personal viewpoints influence (1) news consumption choices, (2) evaluations of news
content, and (3) our ability to critically consume news. The NL tweet in Study 2 similarly
Ac
focuses on audiences but this time frames consumption and evaluation in terms of how to
recognize “fake news,” a term that has gained resonance despite many pitfalls with it (see Funke,
2017). This tweet aims to give users quick tips and actions to take while scrolling through social
media sites; thus, it has a stronger behavioral component than the tweet in Study 1. These studies
aim to evaluate different NL messages to see which are more effective on Twitter.
With the rapidly changing news environment, particularly the growing place of social
media with all its opportunities and drawbacks for news consumption, NL education needs to
also evolve. Experimenting with content, form, and delivery, as we do here, allows for
systematically controlling and testing NL messages.
Study 1 Hypotheses and Research Questions
t
ip
Study 1 uses the topic of genetically modified foods (GMOs) as the misinformation
context and tests NL tweets derived from previous NL research that defines and articulates core
cr
NL concepts (Ashley et al., 2016; Vraga & Tully, 2016). Building on previous research that
us
suggests that exposure to NL messages can alter credibility perceptions of news (e.g., Vraga &
Tully, 2015), we explore perceptions of misinformation on social media and the effects of
an
exposure to an NL tweet:
H1: A control tweet (about planets) will be seen as more credible than a misinformation
M
tweet (about genetically modified foods).
H2: An NL tweet will decrease perceptions of misinformation tweet credibility compared

to the control tweet (about the dangers of texting and driving).
ed
However, because the NL tweets take different forms of address– second-person address (“your
job”) and third-person address (“citizens’ job”) – that have not been previously studied, it is
pt
unclear which will be more effective. Therefore, we ask:

ce
RQ1: Which NL tweet will be more effective at decreasing perceptions of misinformation

tweet credibility as compared to the control tweet?
Finally, considering research suggesting that exposure to NL messages can affect NL

Ac
attitudes and beliefs including self-perceived media literacy and perceptions of the value of
media literacy (e.g., Tully & Vraga, 2018), we propose the following:
H3: NL tweets will boost (a) SPML and (b) VML compared to a control tweet.
RQ2: Which NL tweet will be more effective in boosting (a) SPML and (b) VML?
Study 1 Methods
Participants
An experiment embedded in an online survey was performed in September 2017 and
February 2018.1 1,810 participants were recruited from Amazon’s Mechanical Turk, providing a
diverse but not representative sample of the U.S. population (Levay, Freese, & Druckman,
2016). Participants were paid $1.00 to take a 10-minute survey. Our average participant was 36
t
ip
years old and had a Bachelor’s degree; 50% were female (N=905).
Procedure
cr
For this study, we analyze a 3 (Promoted Tweet: Control, Your Job, Citizen’s Job) x 2
us
(Misinformation: Control, GMO misinformation) between-subjects experimental design.2 All
participants read a simulated Twitter feed with six tweets. We focus on Twitter due to its role as
an
a conduit for misinformation (Ghenai & Mejova, 2017; Starbird, Maddock, Orand, Achterman,
& Mason, 2014).

M
Stimuli
Two tweets were manipulated for this experiment; control tweets were drawn from
ed
previously-validated posts for social and news topics. The first manipulation altered the first
tweet in the feed. Participants saw a “promoted tweet,” the designation Twitter uses for paid
pt
posts, from either the Ad Council about the dangers of texting and driving as a control, or an NL
ce
tweet from the Media Literacy Coalition (a fictitious group) reminding participants that it is
either citizens’ job or your job to recognize how citizens’ (your) viewpoints influence news
Ac
choices and evaluations and encouraging them to be critical news consumers (see Appendix A).
1
Data was collected at two points to ensure adequate power for the analyses performed. We collected data from
1,207 participants in September 2017 and 603 participants in February 2018. We performed a series of t-tests and
chi-square tests to examine differences between the samples, which revealed the February sample significantly more
likely to have a Twitter account (60% vs. 53%), more educated (3.40 vs. 3.21), and wealthier (2.77 vs. 2.64). No
differences in terms of age, party affiliation, ideology, time spent on survey, or misperceptions about GMOs in the
pre-test were observed. We controlled for the fielding date in all analyses.
2
Three conditions were not analyzed for this study. These conditions manipulated whether the misinformation about
GMOs was corrected (1) by an expert, (2) whether that expert correction was endorsed with “favorites” or (3)
whether the correction came from both an expert and a regular Twitter user.
The second manipulation involved misinformation regarding the safety of GMO foods.
The issue of GMO foods was selected because there is scientific consensus that GMO foods are
safe, but the public remains divided (Pew, 2015). In the control condition, a user shared a news
article about planets. In the misinformation condition, a user claimed that scientists “know”
GMOs are unsafe to eat and shared a meme from that claimed that most scientists say GMOs are
t
ip
UNSAFE (Appendix A). Tweets featured a neutral name and picture for the misinformation and
control posts to mitigate the potential for gender biases to skew responses. All analyses are
cr
limited to participants in these six experimental conditions who passed an attention check in the
us
post-test (N=702).3
Study 1 Measures
an
Tweet credibility. Participants were asked to rate the credibility of the manipulated tweet
using a series of semantic differentials on five-point scales, adapted from credibility scales (e.g.,
M
Fico, et al., 2004; Meyer, 1988). Participants rated whether they found the tweet about planets or
GMOs as unbiased, fair, useful, trustworthy, accurate, credible, which were combined into an
ed
index (planets =.83, GMOs =.94).
Self-perceived media literacy. A series of four questions adapted from Vraga et al.
pt
(2015a) asked participants to rate their agreement on five-point scales to measure SPML. These
ce
items were combined into an index ( =.80; see Appendix B).
Value for media literacy. Three questions adapted from Vraga et al. (2015a) were used to
Ac
gauge VML and combined into an index ( =.73; see Appendix B).
GMO misperceptions. In the pre-test, participants’ GMO misperceptions were measured
with two items taken from Bode and Vraga (2015): “GMO foods are safe to eat” [reversed] and
“GMO foods can cause illness in humans” measured on five-point Likert scales, averaged to
3
Attention was measured with an item in the post-test that asked participants to select “somewhat disagree” if they
were paying attention. Participants who selected another option were excluded (N=56).
form an index measuring GMO misperceptions (r=.71, p=.00). These items were obscured in a
battery asking about participants’ attitudes towards vaccinations and climate change.
See Table 1 for descriptive statistics for all Study 1 variables.
Study 1 Results
Message Recall
t
ip
Before turning to our main analysis, we examine message recall for the promoted tweet
to determine if participants recalled seeing the manipulation. After viewing the Twitter feed,
cr
participants were asked if they saw a tweet about eight topics, including “critical news
us
consumption” (NL tweet) and “texting and driving” (control tweet). Only 57% (Your Job,
N=134) and 66% (Citizen Job, N=148) reported seeing the critical news consumption tweet in
an
their respective condition, compared to 85% (N=205) who saw a texting and driving tweet in the
control condition. People in the NL tweet conditions were more likely to report seeing a tweet
M
about “critical news consumption” (p<.001) than in the “texting” condition, and less likely to see
a tweet about “texting and driving” (p<.001; see Table 2). It is also worth noting that seeing the
ed
misinformation tweet (p<.01) compared to the control tweet about planets tended to depress self-
reported exposure to the tweet about “critical news consumption.” Likewise, those with higher
pt
pre-existing GMO misperceptions were less likely to report seeing a tweet about critical news
ce
consumption (p<.05), regardless of condition.
Hypothesis Testing
Ac
All subsequent analyses are limited to those individuals who correctly indicated seeing
the appropriate promoted tweet (N=487). We use a series of two-way ANCOVAs, entering
misinformation condition and promoted tweet topic as factors while controlling for fielding date
and GMO attitudes.

Our first hypotheses tested the effects on credibility assessments of the tweet. As
predicted by H1, we find a main effect of misinformation on credibility (F(1,479)=59.90, p=.00,
partial ƞ2=.111), with the misinformation tweet rated as less credible (M=2.36, S.E.=.06) than
the control tweet (M=2.98, S.E.=.05). However, the data do not support H2, which predicted the
credibility gap between the misinformation and control tweets would be higher for those people
t
ip
who saw an NL tweet. Neither the interaction (F(2, 479)=1.70, p=.18 partial ƞ2=.007), nor the
main effect of promoted tweet topic (F(1, 479)=1.10, p=.37, partial ƞ2=.004) were significant
cr
(RQ1); neither NL message is more effective.
us
Turning to media literacy attitudes, we find partial support for H3, which predicted that
exposure to either NL tweet would boost perceptions of one’s own media literacy (SPML)
an
compared to the control, F(2,479)=3.53, p=.03, partial ƞ2=.015. Specifically, the “Your Job”
tweet led to heightened SPML (M=4.03, S.E.=.06) compared to the control (M=3.82, S.E.=.05,
M
p=.01, with the “Citizen Job” tweet falling between (M=3.95, S.E.=.06), addressing RQ2. In
contrast, we find no significant effects of the NL tweet on VML (F(2, 479)=1.16, p=.31, partial
ed
ƞ2=.005).4
Study 2 Hypotheses and Research Questions

pt
A second study was performed to address some of the shortcomings of Study 1, focusing
ce
on NL message and misinformation context. First, the NL message in Study 1 was derived from
prior research that validated it effectively communicated its core message (Vraga & Tully,
Ac
2016), but did so in a different medium (video), which might have led to a less effective message
on Twitter. To remedy this, in Study 2 we use an NL message more directly designed to combat
the spread of misinformation on social media. This tweet was adapted from an existing tweet
4
We also test the interaction between the presence of misinformation and the NL message to ensure this main effect
is not obscured by an interaction between these two factors. The interaction is not significant for either SPML (F(2,
479)=.17, p=.85, partial ƞ2=.001) or VML (F(2, 479)=1.84, p=.16, partial ƞ2=.008).
from a prominent news literacy organization (News Literacy Project) to reflect the types of NL
messages that are likely to be seen on social media and was pre-tested to ensure audiences
5
accurately perceived its message.
Second, Study 1 tested the impact of an image meme containing misinformation. This
meme had no source information and simply contained a declaration of “truth.” While memes
t
ip
are commonly shared on social media and can contain misinformation, for Study 2 we decided to
use a commonly shared false story from YourNewsWire.com, which incorrectly claimed that the
cr
seasonal flu caused the flu (see Appendix A). This false story had been shared more than 60,000
us
times (Medrano, 2018) and received nearly a half million “engagements” in January 2018,
during the height of flu season (Boland, 2018).

an
Finally, Study 1 focused on an enduring issue – the safety of genetically modified foods –
despite the scientific consensus that GMO foods are safe to eat (Pew, 2015). However, because
M
previous research suggests that corrections work differently for old versus emerging issues (Bode
& Vraga, 2015), in Study 2 we focus on a new issue: the idea that the seasonal flu vaccine is
ed
deadly. While there is again scientific consensus that the flu vaccine is safe and effective, with
both the CDC and AMA strongly recommending all eligible adults get a flu shot, there is
pt
substantial misinformation suggesting the flu shot is unsafe (CDC, 2017).

ce
Therefore, our second study allows us to determine whether more targeted NL messages
may be more effective in increasing skepticism of misinformation on social media in a different

Ac
misinformation context. In Study 2, we replicate the first three hypotheses from Study 1, which
hypothesized that the NL message would increase skepticism of misinformation, improve SPML,
and boost VML, to further examine the mechanisms behind both perceptions of and effects of the
NL tweets. We also examine whether the NL tweet is especially effective in boosting SPML and
5
For additional information on the pretest, please contact the lead author.
VML when combined with a misinformation tweet, which may increase the salience of the
message. Additionally, we ask about the credibility of the story shared and of the site from which
it originated:
H1: A control tweet will be seen as more credible than the flu misinformation tweet.
H2: An NL tweet will decrease perceptions of misinformation tweet credibility compared
t
to a control tweet.
ip
H3: An NL tweet will boost SPML and VML compared to a control tweet.
cr
H4: An NL tweet will decrease perceptions of the credibility of (a) a false story and (b)
the source of the story, compared to a control tweet.
us
Study 2 Methods
Participants
an
The Study 1 design was replicated in Study 2, recruiting 1,214 participants from
Amazon’s Mechanical Turk in February 2018. Participants were paid $1.00 for completing the
M
study. The average participant was 37 years old and had a Bachelor’s degree; 54% were female
(N=656).
ed
Procedure
We analyze a 2 (Promoted Tweet: Control, Recognizing fake news) x 2 (Misinformation:

pt
Control, Flu misinformation) between-subjects experimental design.6 All participants read a

ce
simulated Twitter feed similar to Study 1.
Stimuli
Ac
The first manipulation varied a control tweet on the dangers of texting and driving with
an NL tweet about how to spot fake news. We then manipulated exposure to misinformation
about the flu vaccine. In the experimental condition, a Twitter user posted a false story from
6
There was one correction condition that we did not analyze in this study. These corrections manipulated whether
the misinformation about the flu vaccine was corrected by the American Medical Association (AMA). For more
information on this condition, please contact the lead author.
YourNewsWire.com that claimed the flu vaccine causes the flu and is responsible for flu-related
deaths. The same news tweet about planets from Study 1 was used as the control. All analyses
are limited to those participants in these four experimental conditions who passed the attention
check (N=787; N=51 did not pass the attention check).
Study 2 Measures
t
ip
Tweet credibility. The same measure of tweet credibility from Study 1 was used for the
planets or flu vaccine tweet (planets =.82, flu =.94).
cr
News story credibility. Immediately after asking about tweet credibility, the same six
us
items were used to measure the credibility of the story shared in the planets or flu vaccine tweet,
which were combined into an index (planets =.88, flu =.95).

an
Site credibility. In the post-test, participants were asked to rate the credibility of
YourNewsWire.com on a series of five semantic differentials, rating their trustworthiness,

M
expertise, sincerity, concern for society, and knowledge on five-point scales. These items were
combined to form an index ( =.92).

ed
Self-perceived media literacy. Study 2 replicated the SPML measures from Study 1
( =.82).
pt
Value for media literacy. Study 2 replicated the VML measures from Study 1 ( =.68).
ce
Flu vaccine misperceptions. A single item in the pre-test was used to measure flu vaccine
misperceptions, which asked participants to rate their agreement on a seven-point scale with the
Ac
statement “the flu vaccine causes the flu.” This item was obscured in a battery asking about
participants’ attitudes towards GMO foods and climate change.
See Table 3 for descriptive statistics for all variables measured in Study 2.
Study 2 Results
Message Recall
After viewing the simulated Twitter feed, participants were asked if they saw a tweet
about eight topics, including “news literacy,” and “texting and driving.” We again see an
inequality in reporting seeing the promoted tweet: 88% (N= 346) correctly reported viewing the
“texting and driving” tweet in the control condition, whereas only 64% (N=253) reported seeing
the “news literacy” tweet in the NL condition. Examining participants’ ability to report seeing
t
ip
the promoted tweets, using logistic regression (see Table 4), we find strong main effects of
condition: people in the NL tweet condition were more likely to report seeing a tweet about
cr
“news literacy” (p<.001), but less likely than those in the texting condition to report seeing a
us
tweet about “texting and driving” (p<.001). We also find that people who are misinformed about
the flu vaccine in the pre-test were less likely to report seeing a “news literacy” tweet (p<.001),
an
but there were no effects of seeing a misinformation tweet on seeing either the “news literacy” or
“texting and driving” tweet.

M
Hypothesis Testing
As in Study 1, analyses are limited to those participants who correctly reported seeing the
ed
promoted tweet (N=599). We use a series of two-way ANCOVAs, controlling for pre-test
attitudes towards flu vaccine safety.

pt
Our first hypothesis tested the effects of the NL message and misinformation on
ce
perceptions of the credibility of the tweet itself. We find a strong main effect of misinformation
(F(1, 599)=128.27, p=.00, partial ƞ2=.178), with the tweet containing a link to misinformation
Ac
being seen as much less credible (M=2.17, S.E.=.05) than the control tweet (M=2.92, S.E.=.05).
However, in this case we see a marginal main effect of the promoted tweet (F(1,594)=3.24,
p=.07, partial ƞ2=.005), with the NL tweet leading people to rate the control and misinformation
stories as less credible (M=2.48, S.E.=.05) than when people saw the texting tweet (M=2.60,
S.E.=.04). This main effect is conditioned by a marginal interaction (F(1,599)=3.67, p=.06,

partial ƞ2=.006), supporting H1. In the control condition, the tweet is rated equally credible
regardless of promoted tweet (p=.94); whereas the misinformation tweet was rated as less
credible when viewed with the NL tweet as compared to the texting tweet (p=.01; see Figure 1).
We expand on this hypothesis by exploring the effects of misinformation and NL tweet
on perceptions of the credibility of the story and the site itself (H4). For evaluations of the story,
t
ip
we see the same pattern of effects as for evaluations of the tweet. There is a strong main effect of
the misinformation (F(1, 594)=112.20, p=.00, partial ƞ =.159), with the misinformation story
2
cr
rated as less credible (M=2.25, S.E.=.05) than the control (M=3.01, S.E.=.05). Likewise, there is
us
a significant main effect of the promoted tweet (F(1,594)=3.84, p=.05, partial ƞ2=.006). The NL
tweet led people to rate the stories overall as less credible (M=2.56, S.E.=.06) than the texting
an
tweet (M=2.70, S.E.=.05). The interaction between the NL message and the misinformation is
again marginally significant (F(1,594)=3.62, p=.06, partial ƞ2=.006), with the gap in credibility
M
ratings of the story differing only for the misinformation story (p=.01), but not for the control
story (p=.96; see Figure 2).

ed
This pattern is largely replicated for credibility evaluations of YourNewsWire.com. There
is a main effect of misinformation (F(1,594)=6.47, p=.01, partial ƞ2=.011) – the site is rated as
pt
less credible when people saw the misinformation story from that site (M=2.35, S.E.=.05) than
ce
when they saw a control story (M=2.53, S.E.=.05). But while there is no evidence of a main
effect of exposure to the NL tweet on these evaluations (F(1,599)=.02, p=.90, partial ƞ =.000),
2
Ac
there is a significant interaction between exposure to the NL tweet and the misinformation tweet
(F(1,594)=5.58, p=.02, partial ƞ =.009). The post-hoc analyses suggest that the difference
2
between evaluations of the credibility of the site between the misinformation and control
condition was significant only among those who saw the NL tweet (p=.00), but not among those
who saw the texting and driving tweet (p=.89; see Figure 3). However, in contrast to Study 1, we
find no effects of the NL messages on SPML or VML. Specifically, we do not find a main effect
of NL tweet on SPML (F(1,594)=.02, p=.88, partial ƞ =.000) or VML (F(1,594)=.64, p=.42,

2
partial ƞ =.001).
2 7
Discussion
As calls for increased and improved media literacy to combat “fake news” and
t
ip
misinformation have proliferated (Bulger & Davison, 2018), little empirical work has
investigated how to develop messages designed to be consumed and to be effective on social
cr
media (see Clayton et al., 2019 for an exception). Creating NL messages that resonate with
us
audiences on social media is difficult, as these messages must compete with other content for
audiences’ attention and must prompt critical thinking from audiences as they scroll through
an
posts. The fast-paced nature of social media, in which consumers must evaluate news and
information quickly, presents a particularly challenging environment for making informed news
M
decisions (Head et al., 2018).
Ideally, NL messages on sites like Twitter would encourage people to be more critical
ed
consumers of news and information – encouraging skepticism toward low-quality or false
information and bolstering perceptions of high-quality information. Importantly, these kinds of

pt
interventions are proactive, rather than reactive, and have been shown to be effective at
ce
addressing misinformation and promoting critical engagement with news and information online
(Bolsen & Druckman, 2015; Clayton et al., 2019; Cook, Lewandowsky, & Ecker, 2017; Vraga &
Ac
Tully, 2015). However, our study suggests that it is difficult to alter misinformation perceptions
and NL beliefs with a single message. Although the NL tweets in Study 1 boosted self-perceived
media literacy as expected, they did not increase skepticism of misinformation or boost
7
We again test the interaction between exposure to misinformation and to the NL message. As with Study 1, this
interaction is not significant for SPML, F(1,594)=.56, p=.46, partial ƞ2=.001, or VML, F(1,594)=1.87, p=.17,
partial ƞ2=.003.
perceptions of the societal value of media literacy. In contrast, the NL tweet in Study 2 increased
skepticism of misinformation but did not boost SPML or VML. In short, the NL tweets were
somewhat successful in affecting credibility perceptions and NL beliefs.
In addition, we cannot determine if the effects of exposure to the NL tweets were long-
lasting, however, priming audiences to apply NL knowledge and skills “in the moment” when
t
ip
they are encountering news and information on social media could address both credibility
perceptions and NL beliefs as the tweets examined here were able to alter both, albeit not with
cr
the same message. These results suggest the need for NL campaigns that use multiple tailored
us
and tested messages that are repeated to bolster key messages. As researchers and practitioners
continue to think about ways to improve media literacy, these strengths and limitations should be
an
considered.
Furthermore, the tweets in our studies were consumed in different information contexts.
M
While both topics are scientific issues about which misinformation abounds (CDC, 2017; Pew,
2015), the safety of GMO foods is an enduring issue whereas the specific story about the flu
ed
vaccine is an emerging issue, which may affect the mechanisms by which NL tweets work.
Previous research suggests it may be easier to mitigate misinformation about a relatively new
pt
issue rather than one for which people have already made up their minds (Bode & Vraga, 2015;
ce
Lewandowsky et al., 2012), which may explain the ability of the NL messages to generate
skepticism only for the flu vaccine issue. Research should continue to explore the role of context
Ac
as it has been shown to influence perceptions of NL messages (Tully & Vraga, 2017).
An important and disappointing finding is that the NL tweets did not capture participants’
attention in the same way as the control tweet about the dangers of texting while driving.
Participants correctly reported seeing the control tweet 85% of the time, compared to 62% of the
time for NL tweets in Study 1 and 88% versus 64% in Study 2. This suggests that tweets need to
be designed with social media in mind and also need to be shared regularly to potentially capture
the fleeting attention of Twitter users. Our tweets contained images and hashtags following best
practices, but results suggest that more work needs to be done to create resonant messages
(Adornato, 2017).
In addition, there was inequality in which participants noticed these NL tweets. Across
t
ip
both studies, individuals with greater misperceptions on the issue being studied were less likely
to report seeing the NL tweet. This may suggest a studied ignorance of the tweets as a motivated
cr
reasoning mechanism – rather than rating the NL tweets as not credible to avoid considering
us
their implications, they simply did not recall seeing them (Kunda, 1990). Likewise, participants
in the misinformation conditions were less likely to report noticing the NL tweet in Study 1,
an
which may highlight the difficulty of grabbing attention in a crowded media environment. The
misinformation tweets were also designed to capture attention suggesting that audiences may
M
have been only able to focus on some of the content they consumed. Taken together, these
outcomes suggest that individuals’ predispositions and context matter for processing and
ed
reception of NL messages (Tully & Vraga, 2017).
There are a number of limitations that must be addressed. First, although our samples are
pt
diverse, they are not representative. Importantly, participants were fairly well educated, which
ce
may mean that they process NL messages differently than less educated audiences (Vraga &
Tully, 2016). In addition, we only examine people who accurately recalled the NL tweet. We
Ac
believe such an approach appropriate because we do not expect to perceive effects when
participants do not recall the message of the NL tweets, but this may create inequalities across
conditions and ignores the potential for people to strategically report not seeing an NL tweet that
may counter their predispositions. Finally, and most importantly, because we altered the NL
tweet message and misinformation context, we cannot determine if differences in the tweets’
ability to affect SPML, VML, and misinformation perceptions is a result of the NL tweet or issue
context. We acknowledge that these differences in study design make it more difficult to
interpret the mechanism by which the NL messages operate. However, we think this work
represents an important step toward understanding the ways in which NL messages function on
social media sites like Twitter.
t
ip
Despite these limitations, our findings suggest that NL tweets were able to alter NL
beliefs and misinformation perceptions to some degree. With the ever-present and ever-evolving
cr
problem of misinformation online, we remain hopeful that research will find creative and
us
effective ways to increase news literacy as part of a broader effort to combat the spread of
misinformation.
an
M
ed
pt
ce
Ac
References
Adornato, A. (2017). Mobile and social media journalism: A practical guide. Sage: Thousand
Oaks, CA.
Ashley, S., Maksl, A., & Craft, S. (2017). News media literacy and political engagement: What’s
the connection. Journal of Media Literacy Education, 9(1), 79–98.
t
ip
Bhandari, M. (2018). Social media cues and news site name: What do they mean for online news
perception. Newspaper Research Journal, 39(2), 169–179.
cr
Bode, L., & Vraga, E. K. (2015). In related news, that was wrong: The correction of
us
misinformation through related stories functionality in social media. Journal of
Communication, 65, 619–638. doi: 10.1111/jcom.12166

an
Boland (2018, January 24). What does the Facebook News Feed change really mean for content?
Retrieved from http://www.newswhip.com/2018/01/algorithm-change/

M
Bolsen, T., & Druckman, J. N. (2015). Counteracting the politicization of science. Journal of
Communication, 65, 745–769. doi:10.1111/jcom.12171

ed
boyd, d. (2017, January 5). Did media literacy backfire? Retrieved from
https://points.datasociety.net/did-media-literacy-backfire-7418c084d88d
pt
boyd, d. (2018, March 16). A few responses to criticism of my SXSW-Edu keynote on media
ce
literacy. Retrieved from https://medium.com/@zephoria/a-few-responses-to-criticism-of-
my-sxsw-edu-keynote-on-media-literacy-7eb2843fae22
Ac
Bulger, M., & Davison, P. (2018). The promises, challenges, and futures of media literacy. Data
& Society Research Institute. Retrieved from datasociety.net
Burgoon, M., & Miller, M. D. (1990). Overcoming resistance to persuasion via contiguous
reinforcement and repetition of message. Psychological Reports, 66, 1011–1022.

Center for Media Literacy (n.d.). Media literacy: A definition and more. Retrieved from
https://www.medialit.org/media-literacy-definition-and-more
Centers for Disease Control and Prevention (2017). National early-season flu vaccination
coverage, United States, November 2017. Retrieved from
https://www.cdc.gov/flu/fluvaxview/nifs-estimates-nov2017.htm
t
ip
Clayton, K., Blair, S., Busam, J. A., Forstner, S., Glance, J., Green, G., … Nyhan, B. (2019).
Real solutions for fake news? Measuring the effectiveness of general warnings and fact-
cr
check tags in reducing belief in false stories on social media. Political Behavior. Advance
us
online publication. doi: 10.1007/s11109-019-09533-0
Cook, J., Lewandowsky, S., & Ecker, U. K. H. (2017). Neutralizing misinformation through
an
inoculation: Exposing misleading argumentation techniques reduces their influence.
PLoS One, 12(5): e0175799.

M
Craft, S., Ashley, S., & Maksl, A. (2016). Elements of news literacy: A focus group study of how
teenagers define news and why they consume it. Electronic News, 10(3), 143–160. doi:
ed
10.1177/1931243116656716
Craft, S., Ashley, S., & Maksl, A. (2017). News media literacy and conspiracy theory
pt
endorsement. Communication and the Public, 2, 388–401. doi:

ce
10.1177/2057047317725539
Chu, D., & Lee, A. Y. L. (2014). Media education initiatives by media organizations: The uses of
Ac
media literacy in Hong Kong media. Journalism and Mass Communication Educator, 69,
127–145.
Fico, F., Richardson, J. D., & Edwards, S. M. (2004). Influence of story structure on perceived
story bias and news organization credibility. Mass Communication & Society, 7, 301–
318. doi: 10.1111/j.1083-6101.2006.00322.x

Funke, D. (2017, December 14). Should we stop saying ‘fake news’? Poynter. Retrieved from
https://www.poynter.org/news/should-we-stop-saying-fake-news
Ghenai, A., & Mejova, Y. (2017). Catching Zika fever: Application of crowdsourcing and
machine learning for tracking health misinformation on Twitter. arXiv preprint
arXiv:1707.03778.
t
ip
Head, A. J., Wihbey, J., Metaxas, P. T., MacMillan, M., & Cohen, D. (2018). How students
engage with news. The News Study Report. Project Information Literacy.
cr
Hindman, M., & Barash, V. (2018). Disinformation, “fake news” and influence campaigns on
us
Twitter. Knight Foundation. Retrieved from
https://knightfoundation.org/reports/disinformation-fake-news-and-influence-campaigns-
an
on-twitter
Hofstadter, R. J. (1964). The paranoid style in American politics. Harper’s Magazine.

M
Holan, A. D. (2016). 2016 Lie of the year: Fake news. Politifact. Retrieved from:
http://www.politifact.com/trutho-meter/article/2016/dec/13/2016-lie-year-fake-news/
ed
Kahne, J. & Bowyer, B. (2017). Educating for a democracy in a partisan age: Confronting the
challenges of motivated reasoning and misinformation. American Educational Research

pt
Journal, 54, 3–34. doi: 10.3102/0002831216679817

ce
Kovach, B., & Rosenstiel, T. (2011). Blur: How to know what’s true in the age of information
overload. New York, NY: Bloomsbury Publishing.

Ac
Kunda, Z. (1990). The case for motivated reasoning. Psychological Bulletin, 108, 480–498.
Levay, K. E., Freese, J., & Druckman, J. N. (2016). The demographic and political composition
of Mechanical Turk samples. Sage Open 6(1). doi: 10.1177/2158244016636433.

Levine, P., & Kawashima-Ginsberg, K. (2017). The republic is (still) at risk—and civics is part
of the solution. A Briefing Paper for the Democracy at a Crossroads National Summit.
Medford, MA.
Lewandowsky, S., Ecker, U. K. H., & Cook, J. (2017). Beyond misinformation: Understanding
and coping with the “Post-truth” era. Journal of Applied Research in Memory and
t
ip
Cognition, 6, 353-369.
Lewandowsky, S., Ecker, U. K. H., Seifert, C. M., Schwarz, N., & Cook, J. (2012).
cr
Misinformation and its correction: Continued influence and successful debiasing.
us
Psychological Science in the Public Interest, 13, 106-131. doi:
10.1177/1529100612451018
an
Marwick, A. E. (2018). Why do people share fake news? A sociotechnical model of media
effects. Georgetown Law Technology Review, 474(2), 474–512.

M
Marwick, A. E., & Lewis, R. (2017). Media manipulation and disinformation online. Data &
Society Research Institute. Retrieved from datasociety.net

ed
McGrew, S., Breakstone, J., Ortega, T., Smith, M., & Wineburg, S. (2018). Can students
evaluate online sources? Learning from assessments of civic online reasoning. Theory &
pt
Research in Social Education, 1–29. Advance online publication. doi:

ce
10.1080/00933104.2017.1416320
Medrano, K. (2018). Facebook spreads viral fake news story about vaccines. Newsweek.
Ac
Retrieved from http://www.newsweek.com/facebook-spreads-viral-fake-news-story-
about-vaccines-791331
Metzger, M. J., Flanagin, A. J., & Medders, R. B. (2010). Social and heuristic approaches to
credibility evaluation online. Journal of Communication, 60, 413–439.

Meyer, P. (1988). Defining and measuring credibility of newspapers: Developing an index.
Journalism Quarterly, 65, 567–588. doi: 10.1177/107769908806500301.
Mihailidis, P., & Thevenin, B. (2013). Media literacy as a core competency for engaged
citizenship in a participatory democracy. American Behavioral Scientist, 57, 1611–1622.
doi:10.1177/0002764213489015
t
ip
Mihailidis, P., & Viotty, S. (2017). Spreadable spectacle in digital culture: Civic expression, fake
news, and the role of media literacies in a “post fact” society. American Behavioral
cr
Scientist, 61, 441–454. doi: 10.1177/0002764217701217
us
Nyhan, B., & Reifler, J. (2010). When corrections fail: The persistence of political
misperceptions. Political Behavior, 32, 303–330.

an
Pew Research Center. (2015, January 29). Public and scientists’ views on science and society.
Retrieved from http://www.pewinternet.org/2015/01/29/public-and-scientists-views-on-

M
science-and-society/
Pfau, M., Kenski, H. C., Nitz, M., & Sorenson, J. (1990). Efficacy of inoculation strategies in
ed
promoting resistance to political attack messages: Application to direct mail.
Communication Monographs, 57, 25–43.

pt
Starbird, K., Maddock, J., Orand, M., Achterman, P., & Mason, R. M. (2014). Rumors, false
ce
flags, and digital vigilantes: Misinformation on Twitter after the 2013 Boston Marathon
bombing. IConference 2014 Proceedings.

Ac
Thorson, K., Vraga, E. K., & Ekdale, B. (2010). Credibility in context: How uncivil online
commentary affects news credibility. Mass Communication & Society, 13, 289–313.
Tully, M., & Vraga, E. K. (2017). Effectiveness of a news media literacy advertisement in
partisan versus nonpartisan online media contexts. Journal of Broadcasting & Electronic
Media, 61, 144–162.

Tully, M. & Vraga, E. K. (2018). A mixed methods approach to examining the relationship
between news media literacy and political efficacy. International Journal of
Communication, 12, 766–787.
Tully, M., Vraga, E. K., & Smithson, A. B. (2018). News media literacy, perceptions of bias, and
interpretation of news. Journalism: Theory, Practice, and Criticism. Advance online
t
ip
publication. doi: 10.1177/1464884918805262
Vraga, E. K., & Bode, L. (2018). I do not believe you: how providing a source corrects health
cr
misperceptions across social media platforms. Information, Communication & Society,
us
10, 1337–1353.
Vraga, E. K., & Tully, M. (2015). Media literacy messages and hostile media perceptions:
an
Processing of nonpartisan versus partisan political information. Mass Communication
and Society, 18, 422–448.

M
Vraga, E. K., & Tully, M. (2016). Effective messaging to communicate news media literacy
concepts to diverse publics. Communication and the Public, 1, 305–322.

ed
Vraga, E. K., Tully, M., Kotcher, J. E., Smithson, A. B., & Broeckelman-Post, M. (2015). A
multi-dimensional approach to measuring news media literacy. Journal of Media Literacy

pt
Education, 7(3), 41–53.

ce
Ac
Tables and Figures
Table 1: Descriptive statistics for Study 1 variables
Control promoted tweet Your Job promoted tweet Citizens Job promoted tweet
Control Misinformation Control Misinformation Control Misinformation
N=111 N=94 N=75 N=59 N=77 N=71
M SD M SD M SD M SD M SD M SD
Tweet 3.07 .74 2.52 1.17 3.00 .73 2.21 1.00 2.83 .67 2.36 1.11
credibility
t
SPML 3.86 .77 3.74 .88 4.07 .59 4.03 .73 4.00 .73 3.90 .70
ip
VML 4.30 .64 4.23 .68 4.30 .61 4.47 .55 4.29 .61 4.24 .60
Pre-test 2.75 1.07 2.66 1.17 2.45 1.10 2.41 1.16 2.42 1.19 2.73 1.12
misperceptions
cr
Table 2: Logistic Regressions for Message Recall in Study 1
us
Critical News Consumption Texting and Driving
B S.E. Odds B S.E. Odds
ratio ratio
Your Job 3.35 .34 28.52*** -5.23 .42 .005***
an
Citizen Job 3.78 .34 43.69*** -6.58 .74 .001***
Misinformation -.56 .19 .57** -.50 .33 .61
GMO misperceptions -.22 .08 .80* .03 .15 .85
Field -.15 .20 .86 -.65+ .34 .52
M
Cox and Snell R-square .316 .562
Nagelkerke R-square .426 .793
ed
Table 3: Descriptive statistics for Study 2 variables
Control promoted tweet Fake News promoted tweet

Control Misinformation Control Misinformation
N=168 N=178 N=128 N=125
pt
M SD M SD M SD M SD
Tweet 2.97 .74 2.29 1.07 2.89 .68 2.00 .86
credibility
ce
SPML 3.89 .82 3.93 .74 3.95 .73 3.91 .70

VML 4.16 .64 4.16 .61 4.20 .60 4.32 .52
Pre-test 3.27 1.69 2.39 1.10 2.85 1.64 2.77 1.48
misperceptions
Ac
News story 3.07 .79 2.43 .95 2.98 .76 2.07 .97
credibility
Website 2.48 .86 2.99 1.74 2.60 .83 2.24 .86
credibility
Table 4: Logistic Regressions for Message Recall in Study 2
News Literacy Texting and Driving

B S.E. Odds B S.E. Odds
ratio ratio
NL tweet 4.42 .36 83.08*** -6.35 .48 .002***
Misinformation -.13 .20 .88 .05 .29 1.05
Flu vaccine misperceptions -.24 .06 .79*** .02 .09 1.02
t
Cox and Snell R-square .411 .610
ip
Nagelkerke R-square .571 .816
cr
us
an
M
ed
pt
ce
Ac
Figure 1: Effects on tweet credibility (Study 2)
t
ip
cr
us
Figure 2: Effects on story credibility (Study 2)
an
M
ed
pt
Figure 3: Effects on YourNewsWire.com credibility (Study 2)

ce
Ac
Appendix A: Stimulus tweets

Neews Literaccy Messagess for Social Media 32
Study 1 NL “Your JJob” tweet Stuudy 2 NL tweeet
t
ip
cr
us
Study 1 misinformaation tweet Stuudy 2 misinfo
an formation tw
weet
M
ed
pt
ce
Ac
Appendix B: NL Measures
Self-perceived media literacy (SPML):

• I have a good understanding of the concept of media literacy
• I have the skills to interpret media messages
• I understand how news is made in the US
• I’m not sure what people mean by media literacy (reversed)
Value for media literacy (VML):
t
• People should understand how media companies make decisions about news content
ip
• The news media have a role to play in informing citizens about civic issues
• Media literacy is important to democracy
cr
us
an
M
ed
pt
ce
Ac

Designing and Testing News Literacy Messages For Social Media PDF

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Designing and Testing News Literacy Messages For Social Media PDF

Uploaded by

Copyright:

Available Formats

Mass Communication and Society

ISSN: 1520-5436 (Print) 1532-7825 (Online) Journal homepage: https://www.tandfonline.com/loi/hmcs20

Designing and Testing News Literacy Messages for

Melissa Tully, Emily K. Vraga & Leticia Bode

To link to this article: https://doi.org/10.1080/15205436.2019.1604970

Accepted author version posted online: 11

Submit your article to this journal

View Crossmark data

Full Terms & Conditions of access and use can be found at

Journal: Mass Communication and Society

campaigns that feature multiple messages and calls to action.

Keywords: News literacy, misinformation, credibility, self-perceived media literacy, Twitter

Designing and Testing News Literacy Messages for Social Media

and misleading information making it particularly suitable to addressing the spread of

encourage critical news consumption to mitigate the impact of exposure to misinformation, as

Misinformation and Credibility

Misinformation is commonly defined as “objectively incorrect information” (Bode &

in particular boosting credibility perceptions of accurate information and decreasing credibility

perceptions of misinformation – might be a mechanism by which rejecting misinformation

evaluate the promotion of critical evaluation of information as a mechanism for assessing

information quality and affecting credibility perceptions.

NL efforts seek to change perceptions of information credibility – or people’s judgments

Edwards, 2004; Meyer, 1988) – ideally bolstering credibility perceptions of high-quality

information and decreasing perceptions of low-quality or false information. The question of

analyzing news stories (Tully et al., 2018).

Research suggests that exposure to NL messages should affect perceptions of information

exposure to NL messages should affect perceptions of information credibility in two ways

recognize poor information and acknowledge it as less credible.

need for continued efforts to develop effective messages.

NL messages designed to be consumed as part of regular news consumption on social

systematically controlling and testing NL messages.

Study 1 Hypotheses and Research Questions

H2: An NL tweet will decrease perceptions of misinformation tweet credibility compared

unclear which will be more effective. Therefore, we ask:

RQ1: Which NL tweet will be more effective at decreasing perceptions of misinformation

Finally, considering research suggesting that exposure to NL messages can affect NL

An experiment embedded in an online survey was performed in September 2017 and

& Mason, 2014).

index (planets =.83, GMOs =.94).

items were combined into an index ( =.80; see Appendix B).

GMO misperceptions. In the pre-test, participants’ GMO misperceptions were measured

See Table 1 for descriptive statistics for all Study 1 variables.

consumption (p<.05), regardless of condition.

and GMO attitudes.

predicted by H1, we find a main effect of misinformation on credibility (F(1,479)=59.90, p=.00,

Study 2 Hypotheses and Research Questions

during the height of flu season (Boland, 2018).

substantial misinformation suggesting the flu shot is unsafe (CDC, 2017).

may be more effective in increasing skepticism of misinformation on social media in a different

H2: An NL tweet will decrease perceptions of misinformation tweet credibility compared

We analyze a 2 (Promoted Tweet: Control, Recognizing fake news) x 2 (Misinformation:

Control, Flu misinformation) between-subjects experimental design.6 All participants read a

simulated Twitter feed similar to Study 1.

check (N=787; N=51 did not pass the attention check).

planets or flu vaccine tweet (planets =.82, flu =.94).

which were combined into an index (planets =.88, flu =.95).

YourNewsWire.com on a series of five semantic differentials, rating their trustworthiness,

combined to form an index ( =.92).

participants’ attitudes towards GMO foods and climate change.

“texting and driving” tweet.

attitudes towards flu vaccine safety.

S.E.=.04). This main effect is conditioned by a marginal interaction (F(1,599)=3.67, p=.06,