Introducing the Visual Conjoint, with an

Application to Candidate Evaluation on Social

Media
Alessandro Vecchiato Kevin Munger

July 15, 2021

Abstract
Conjoint experiments have seen an explosion of interest within political sci-
ence, enabling scholars to understand the preferences of citizens in a variety
of political contexts. However, one notable limitation of the standard conjoint
design is the textual medium; citizens frequently encounter political informa-
tion encoded as images, and in particular, encounter implicit cues that cannot
be unawkwardly translated into the literal form of that standard conjoint. We
present a method for algorithmically generating images at scale, enabling us de-
ploy conjoint experiments where subjects select between two images that encode
the denotational information possible in standard conjoints in addition to the
subtle image- or style-based information prominent in more ecologically valid
encounters with political information. We explore the utility of this method
in an application where subjects evaluate the Twitter profiles of hypothetical
candidates. We demonstrate that the visual conjoint allows subjects to take in
image-based information and that it can also incorporate the social endorsement
information that is central to politics on social media.

1
1 Introduction

Conjoint experiments have recently made a significant impact on the study of voter
preferences. Most notably, researchers have used conjoint experiments to study how
candidate’s attributes such as age or race influence voters’ support (Hainmueller,
Hopkins, and Yamamoto, 2014a).

In a typical conjoint, respondents are presented with randomly generated pro-

files of candidates taking the form of a list of features (age, race, political party),
and asked to pick or rank the choices based on their preferences. Such a design is
not ideal for representing social media profiles, for which much of the experience is
visual and contextual. Furthermore, the standard conjoint layout implies an equal
weight afforded to each facet, an artifact of the design that rarely occurs in the every-
day practice of electoral politics.

We develop, instead, a form of visual conjoint that mimics an ordinary conjoint in

terms of the full randomization of features but presents the profiles in the form of
an image. In the current application, our visual conjoint takes the form of a Twitter
profile. The design is flexible, however, and could be applied to contexts including po-
litical advertisements, synthetic images of politicians, and partisan stereotypes. The
task of evaluating a politician (or, indeed, a potential dating partner or employee)
based on their social media profile is an increasingly common part of political, social
and economic life; we argue that the visual conjoint like the one we deploy here is
higher in ecological validity than the standard conjoint.

Our design includes typical details for candidate studies like age, race, gender, and

2
education, allowing us to benchmark our experiment against similar experiments run
with standard conjoints. But the social media case affords the capacity to compare
the magnitude of these well-established drivers of candidate preference with crucial
elements specific to the social media that inflect real-world citizen evaluations. Here,
we investigate the importance of social feedback, in the form of the number of followers
of the hypothetical politician profiles and in the number of “likes" and “retweets" on
a handful of tweets included as part of the design.

Over the first half of 2021, we deployed the visual conjoint in a candidate prefer-
ence experiment based on the now-classic conjoint in Hainmueller, Hopkins, and Ya-
mamoto (2014b). We recruited 4,194 respondents through Lucid Theorem, a polling
company that has been shown to contain sufficient variation in relevant demograph-
ics. This large sample size allows us to test for the existence of treatment effect
heterogeneity in these theoretically relevant subject characteristics, namely age, dig-
ital literacy, and partisan identity.

Our topline results largely replicate the findings from the original conjoint ex-
periment, suggesting a baseline plausibility for our approach. Once again, we find
that the largest effect is a penalty for politicians with the profession of “car dealer."
We find only slight evidence that the politician characteristics made more salient in
the natural Twitter layout — candidate race and gender (signalled through a cartoon
avatar) — have an amplified effect compared to the standard conjoint, but we suggest
future manipulations which could take fuller advantage of this feature.

The effects of adding social feedback, however, offer the strongest evidence for
the value our approach offers beyond the standard conjoint. Despite evidence of the

3
importance of social endorsements like “Likes" and “Retweets" in how social media
users evaluate the credibility of news stories (Messing and Westwood, 2014; Munger,
2019) and the importance of “follower counts" in how users evaluate the status of
Twitter users (Munger, 2017, 2020), the implications of the centrality of this affordance
of social media platforms has not been fully incorporated into research on their effects.

We find null results for this facet in the full sample; however, this null effect is
masking offsetting treatment effect heterogeneity in the age of the respondents. Re-
spondents in the Baby Boomer generation or older have a significant preference for
politicians with high social feedback; younger respondents have a significant prefer-
ence against them.

These kind of offsetting treatment effects heterogeneities are rare, and our evi-
dence is far from the final word on the topic. But we argue that this intersection of
digital media and subject generation is where we are most likely to find these kind
of large heterogeneities, and hope for more research on the point.

In contrast, we expected analogous heterogeneities along a related dimension:

subject digital literacy. However, operationalizing this construct in either of two dis-
tinct survey batteries yields robustly null results.

Our approach can be extended to study other features of visual political com-
munication especially for social media. For instance, in a separate study we plan
to evaluate the importance of tweets content and communication style. We offer
resources for other researchers to implement the replication materials that include
the Python scripts to generate the conjoint Twitter profiles and deploy the survey

4
instrument in Qualtrics.

2 Conjoint Experiments

Conjoint experiments have recently made a large impact on the study of voter pref-
erences for their theoretical import and easy interpretation of the results. Quick and
efficient in their implementation, conjoint experiments ask respondents to select their
preferred candidate among a portfolio (or a pair) of potential candidate’s profiles car-
rying different demographic characteristics and expressing various policy positions.

The foundational paper in the development of this method in Political Science is

Hainmueller, Hopkins, and Yamamoto (2014b) which includes the theoretical deriva-
tion of the estimands and an application to the candidate selection problem. In the
ensuing years, conjoints have been applied to a broad range of evaluations, including
those of immigrant, neighborhoods, and climate-related policies; see Bansak et al.
(2021) for an overview. In this work, instead, we shy away from the substantive
import of conjoint experiments and engage with an emerging literature about their
methodological aspects to improve the validity of the inferences made from them.
Abramson, Koçak, and Magazinnik (2019), for example, demonstrates the limita-
tions in the range of research question to which conjoints are applicable. Because of
our substantive interest in democratic elections, political scientists would like to be
able to learn about the effect of politician attributes on electoral outcomes. However,
the aggregation of average preferences can produce misleading results if a minority
of the population has strong preferences about a given attribute, adding concerns
about conjoint analysis.
Furthermore, the number of attributes included in the conjoint experiment has

5
implications for the parameter estimates. Bansak et al. (2017) extends the analysis to
the breaking point and evaluates the attenuation bias on parameter estimates result-
ing from the addition of attributes. They find a small but not negligible attenuation
bias.
Relatedly, Egami and Imai (2018) demonstrates the existence of cases in which the
levels of different attribute interact, making the AMCE, the standard estimand for
conjoint experiments, a potentially misleading statistic.

Other methodological concerns relate to implementation itself. The standard con-

joint experiment presents each piece of information as text, filled into a small box.
The subject scans each of the two columns of boxes that comprise the two candidate
profiles and then making a choice (Jenke et al., 2019).

While the artificiality of this research design is not a drawback for most appli-
cations, for the study of candidate’s social media profiles in which features have
different levels of salience, it likely reduces the ecological validity of the results.
That is, as Jenke et al. (2019) demonstrates, respondents pay differential amounts of
attention to different boxes in the standard conjoint; if the distribution of attention
would be different in a more realistic context like evaluating a social media profile,
we argue that the latter design is a more ecologically valid test of how voters evaluate
politicians.

Our concern is related to—but distinct from—the challenge to the external va-
lidity of conjoints raised by De la Cuesta, Egami, and Imai (2019), who argue that
the use of a balanced a distribution of attribute levels can produce misleading infer-
ences about the population. They advocate for population-level distribution of the

6
attributes other than the dimensions of primary interest, in order to maximize the
information gained from the closest possible counter-factual. The uniform distribu-
tion of attribute levels, they argue, is an unnecessary artifice motivated primarily by
convenience. We argue that the uniform distribution of emphasis in the visual design
of the conjoint is a similarly unnecessary artifice, one upon which we can improve by
looking to the distribution of relevant real-world contexts.

For this purpose, we introduce a new format for conjoint experiments that re-
places the standard box-columns with a visual form that people frequently encounter
in their standard experience of using the internet to learn about politicians: the social
media profile.

Visual conjoints have been in use for decades in fields like marketing and en-
gineering (Sylcott, Orsborn, and Cagan, 2016).1 Indeed, these fields have pushed
the modality of conjoint analysis beyond simple images to include interactive sto-
ryboards (Intille, Kukla, and Ma, 2002), virtual reality (Dijkstra and Timmermans,
1998), and even physical products like water bottles (Dominique-Ferreira, Boubeta,
and Mallou, 2012). We invoke these examples to emphasize the non-novelty of the
idea of a visual conjoint.2
Many of these applications, however, require that each of the unique combinations
of the dimensions of interest be created by hand, severely limiting the number of these
attributes and the number of values that each can take. Our approach allows for us
to generate hundreds or even thousands of unique images automatically and embed
1Text-based conjoints have been in use for far longer, suggesting that their relatively recent popu-
larity in Political Science is a fruitful example of disciplinary cross-pollination.
2Another important example from political science is McClean and Ono (2020), who use advanced
photo editing technology to manipulate the apparent age of politicians, appending these images to
standard conjoints. They find that younger people have a revealed preference for co-generationalists
that is largely explained by inferred issue emphasis.

7
them in a Qualtrics survey. The design of the visual elements of the composite image
takes more care than simply plugging in different textual values into a standard
conjoint, but this step only has to be taken once for each visual conjoint experiment.

2.1 Creating the Visual Conjoint

We generated the random Twitter profiles with the Python Image Library. The script
operates in two phases. It first generates generic and empty Twitter profiles that
resemble actual ones in every aspect including color scheme and font selection. Such
profiles report all the same name and handle, Congressman Smith (Congresswoman
for women profiles), they are all localized in Washington, DC, and they all report the
same generic website, congress.gov.

The script then fills the generic profiles through a nested loop over the feature
sets, such as gender and education. Each profile combination picks one element of
each character vector to generate the final profile.
The script operates differently for features signaled through images and those re-
ported through text. For the profile picture avatars that signal both gender and race,
the script reshapes and crops previously generated images and combines them on
the empty profile. For the textual information, the script combines feature levels as
character vectors into a readable Twitter bio.

To avoid specific-level treatment effects, numeric information such as year of birth

(for age) or social media feedback is randomly drawn from specific ranges.
Years of birth in the older generation are a random draw between 1940 and 1975,
while those in the younger generation are from 1976 to 1990.
Social media feedback is divided in two categories, low and high, encompassing

8
all possible feedback elements such as followers and like, and corresponding to
markedly different levels of social media engagement. For instance, low social media
engagement corresponds, among others, to a following between 2 and 5 thousand
people, while such figures for high engagement are between 10 and 13 thousands.
Similarly, Low engagement corresponds to a range between 100 and 800 likes for each
post, while likes on high engagement correspond to 3000 or 6000 likes.

As for any unweighted conjoint experiment, some profiles are implausible. For
instance, it is impossible for a candidate to be only a high-school graduate and work
as a medical doctor. Following their assessment, the researcher can remove the
implausible candidate profiles that have been generated. We act conservatively by
removing only profiles that are implausible in the education-profession combination,
such as lawyers with only high-school diploma.

While the script is specific to the generation of Twitter profiles, it can be readily
adapted to the study of other features of Twitter profiles by modifying existing fea-
ture sets, or adding new ones.

Profiles are then included in a Qualtrics questionnaire that offers them in pairs
from the pool of all profiles generated.

2.2 Application of the Visual Conjoint to Social Media

The “social media visual conjoints" we develop below communicate all of the in-
formation used in a standard conjoint in a more organic manner. The standard
demographic details are encoded in the “bio" field of the account, and the issue posi-
tions are expressed in the few tweets we include with the preview. Figure 1 provides

9
a visual overview. We opted to use Twitter as the social media platform for this
experiment as it is the locus of contemporary political discussion, but the framework
is flexible and social conjoints based on other platforms are possible as well.

In addition to the improved ecological validity of this format, the social media
conjoint allows the researcher to manipulate aspects of the hypothetical candidate
that cannot be communicated through the objective, scientific aesthetic of the stan-
dard conjoint. One example (which we do not pursue in the current experiment)
of the information the social conjoint can communicate is candidate self-presentation;
by using hypothetical candidates’ tweets, these images convey the writing style, the
tone, and the political “manners" in a nuanced way that would be harder to commu-
nicate by denotational text (Goffman et al., 1978).

On the other hand, there are certain elements of the standard candidate-preference
conjoint that cannot be easily communicated in our design. For example, it would be
unnatural for a politician’s Twitter bio to contain an explicit description of how much
money they earn or how old they are, both of which are features manipulated in
Hainmueller, Hopkins, and Yamamoto (2014b). Again, we interpret this as an advan-
tage of our approach: voters encountering politicians through their self-presentation
on social media is increasingly central to electoral politics, so the kind of information
that is naturally conveyed in these contexts should be given more weight by ecological-
validity-minded researchers.

10
3 Experimental Design

Respondents were asked to select which candidate they would prefer between two
randomly selected candidates among 4,800 possible combinations. Respondents
were introduced to the experiment with the following message: "In this section, we
will show you a series of Twitter profiles of hypothetical Members of the US Congress.
These profiles (and the people they represent) are not real, but are constructed from
composites of real people and profiles. For each pair of profiles, please take your
time and read each of the profiles carefully, then tell us which politician you’d prefer
to be your representative in Congress." and then they were were asked: "Which of
these politicians would you rather have as your Representative in Congress?"
Each respondent would have five candidate-pair (tasks) to evaluate. The candidate
profiles varied randomly along the following dimensions: generation, gender, race,
social feedback, education, party, profession, military service and religion. Table 1
lists all possible feature levels for all of the nine dimensions. To avoid confusing
participants with ethnically diverse names, we included identical last names and no
first names (Edwards and Caballero, 2008).

3.1 Recruitment and Selection

To implement the experiment, we follow the recommendation in Munger et al. (2021):

because we hypothesize that the treatment effects of interest will be heterogeneous in
respondent age and digital literacy, we recruited subjects from Lucid Theorem. This
is a standard source of subjects for digital survey experiments, the results of which
have been shown to generalize to nationally representative samples (Coppock, 2018),
but as demonstrated in Guess and Munger (2020), these subjects skew young and
high in digital literacy.

11
 Figure 1: An example of our conjoint task with Two Hypothetical Twitter Profiles

The figure shows two random social media profiles as they would appear to respondents. On the
left, Candidate A is characterized by the following features: Black, Woman, Millennial, with High
Feedback, a College Graduate, and a Teacher (no religion, and no military service). On the right,
Candidate B is characterized by the following features: White, Man, Boomer generation, with High
Feedback, Served, a College graduate working as a Doctor and Protestant. In the experiment, we only
vary college ranking but do not considerate graduate studies.

12
Table 1: Dimensions of Candidate Characteristics

Dimensions Possible Values

Generation Boomer
Millennial
Gender Male
Female
Race White
Black
Feedback High
Low
Party Democrat
Republican
Military Served
Not Served
Education Ivy League
College
No College
Religion Protestant
Catholic
Mormon
No Religion
Jewish
Profession Used Car Dealer
Farmer
Lawyer
Doctor
Teacher

13
 Lucid Theorem is a polling platform that aggregates and samples from several
polling companies to generate samples that are nationally representative according
to standard demographic cross-tabs. Recently, however, there have been concerns
about decline levels of subject attentiveness on the platform (Aronow et al., 2020).

To ensure the quality of our data, we include in the participants’ on-boarding three
attention checks, two after the consent form and another one in the demographic
characteristics section. After the consent form, respondents were asked to answer
two questions structured to instruct respondents on the right answer. The first
question instructed the respondents to choose the answer “I have a question," the
third option after “I understand" and "I do not understand". Mimicking the format of
many initial questionnaire questions, such structure was specifically chosen to screen
out respondents that would try to move forward quickly by clicking “I understand."
The second attention check replicated the structure of the first one, but replacing
the three option with nine options describing common public policy issues, such
as taxes and climate change. In this case, respondents were asked to select the
seventh option (“Climate Change") among the nine options proposed. Screened out
respondents were then redirected back to the Lucid platform and not counted for the
demographic quotas.
Finally, in the demographic characteristics’ portion of the questionnaire, we in-
cluded a third attention-check question asking the respondents to answer, in an open
format, with the number of words contained in the question statement. To avoid
concerns about possible post-treatment bias, we present all results including just
subjects who passed this final attention check as well as the full sample.
Our final sample is of 4,194 respondents for the Lucid sample, 2,570 of whom
passed the attention check and took more than five minutes to complete the survey.

14
3.2 Demographic Characteristics

Before the experimental treatment, respondents were asked a battery of descriptive

questions aimed at gauging, outside of the usual demographics, their level of digital
literacy and their social media use. The conjoint design does not allow for extensive
examination of the effect of respondent characteristics, but our theoretical expecta-
tions pointed towards the importance of two possible dimensions of treatment effect
heterogeneity: digital literacy and age.
Following the recommendations of Guess and Munger (2020), we deployed two
measures of digital literacy. We first used the “internet skills" scale developed by
Hargittai (2009), asking respondents their level of familiarity with fourteen technical
term that appear normally in matter related to the internet and digital life, such
as "cache," "advanced search," and "tagging." Respondents could give a self-rating
ranging from 1 to 5, where 5 corresponded to full understanding of the term. Sec-
ond, we used the “power user" scale developed by Sundar and Marathe (2010) to
gauge respondents’ relationship with internet technology by asking them how much
they agreed with statements describing the digital life, such as "I have problem with
viruses and malware on my computer," and "I feel that information technology is
part of my daily life."
The full questionnaire is presented in the Appendix A.

3.3 Empirical Strategy

Following our design, the empirical strategy is straightforward. We structure the

data so that, for each respondent, it includes as dummy variables the descriptive
characteristics of the profiles selected and seen. We calculate the Average Component

15
Marginal Effect through a simple regression of the form:

Õ Õ Õ
𝑌𝑖𝑗 𝑘 = 𝜃0 + 𝜃𝐿1 𝑋𝐿1,𝑖𝑗 𝑘 + 𝜃𝐿2 𝑋𝐿2,𝑖𝑗 𝑘 + · · · + + 𝜃𝐿9 𝑋𝐿9,𝑖𝑗 𝑘 + 𝜖 𝑖𝑗 𝑘 (1)
𝑙:1..𝐿1 𝑙:1..𝐿2 𝑙:1..𝐿9

for subject 𝑖 evaluating profile 𝑗 in task 𝑘, and where 𝑋𝐿1 are the levels of each of
the nine categories 𝐿1, . . . , 𝐿9 features (excluding the reference one).
For examining the relevant subgroup preferences, we follow Leeper, Hobolt, and
Tilley (2020) and use their R package ‘cregg’ to estimate the difference in AMCEs
in a principled fashion. We also present relevant results that estimate instead the
difference in marginal means, which they argue may be more appropriate for making
causal claims about the differential treatment effects of attributes on subgroups.
The next section presents the results from the analysis.

4 Results

Figure 2b presents Average Component Marginal Effect (ACME) with 95% confidence
intervals for feature levels indicating the nine categories. Effects are relative to the
excluded level for each category, represented in the graph as a dot on the Y-axis.
ACME show the increase in population probability that a profile would be chosen if
its level would change to the one under consideration, averaged over all the possible
values of the other components. The ACME can be identifies by virtue of the random
assignment of each component to profiles (Hainmueller, Hopkins, and Yamamoto,
2014b).

The top panel displays the results on the entire sample of people who passed the
preliminary attention checks, a total N of 4,194. The bottom panel restricts the sample

16
 Figure 2: Preferences for Twitter profiles
(a) Full sample (N = 4,194)

(b) Attention checks (N = 2,570)

The figure reports combined plots for estimates of the change in probability of selecting the candidate,
based on the reported characteristics. Estimates are to be evaluated as averages over all the possible
combinations of the other components. Values on the Y-axis represent the reference category.

17
to individuals who passed the mid-survey attention check and who took more than
five minutes to complete the survey, removing individuals who sped through the
survey and were thus unlikely to have taken up the treatments. The results are very
similar and the inferences identical.
We replicate several of the main findings in Hainmueller, Hopkins, and Yamamoto
(2014a), providing support for the general exercise. Among the denotational infor-
mation written out in the “bio" of the profile, we replicate the finding that the single
largest effect is a large penalty for car dealers. We also replicate the preference for
younger rather than older politicians, and for military veterans. Unlike in the original
survey, we do not find evidence of a penalty for Mormons or Protestants (although
we did not break the latter category into “Mainline" and “Evangelical"). We also
find no effects of education, while the original paper found a penalty for politicians
lacking a college degree.

Politician race and gender, the information conveyed by the image—and thus, in
our design, more salient than the denotational information in the bio—is not signifi-
cant here, just like in the original experiment. We do note that there is a significant
positive selection effect (p = .07) for female politicians, suggesting the possibility that
this information is in fact more salient in our design.

Finally, we find no effect of the social feedback communicated in the follower

counts and the likes and retweets on the accounts. However, this is the effect that we
expected to be heterogeneous in respondent age and digital literacy.
Figures 3b and 4b report these results. In Figure 3b, we find evidence that the
null effect of social feedback in the overall sample is in fact masking offsetting effects
among different generations: Baby Boomers (fifty-seven or older) are more likely

18
to support the politician with more social feedback, while younger respondent are
less likely to support that politician. Both of these effects are significantly different
from zero in the attention check sample (bottom panel); in the full sample, only the
younger-generation penalty is significant, but the AMCE of social feedback is still
significantly moderated by subject age.
In contrast, 4b demonstrates very little effect heterogeneity in subject digital liter-
acy, whether operationalized with the internet skills battery (top panel) or the power
user scale (bottom panel). We only report the results on the full sample, but we find
null results on the attention check sample as well.
Finally, in order to generate a benchmark of the magnitude of treatment effect
heterogeneity, we calculate the effect of partisanship among subjects of different
party affiliations. In Figure 5b we find significant political patterns across parties.
Using the 10-point partisanship scale provided by Lucid, we code respondents
as Republicans if they self-reported being [Other - leaning Republican; Not very
strong Republican; Strong Republican; Independent - Republican], and analogously
for Democrats. The held out category reported being either Independent or Other,
and did not lean towards either major party if prompted.
Using a broad definition for each partisan group, we see the expected hetero-
geneity based on candidate partisanship. The magnitude of this effect has a point
estimate of approximately .07 in each direction, suggesting a large but not dispositive
role for partisan cues. This effect is nearly identical across the two samples. For com-
parison, this is roughly double the magnitude of the treatment effect heterogeneity
in respondent age of social feedback estimated in Figure 3b.
Another heterogeneity in partisanship emerges from the data. There are no
significant differences between Republicans and Democrats in preferring white or
black people, military service, or men or women, but candidate education does seem

19
 Figure 3: Preferences for Twitter profiles, by respondent generation
(a) Lucid, full sample (N = 4,194)

(b) Lucid, attention checks (N = 2,570)

The figure reports OLS estimates of the change in probability of voting for candidate, based on the
reported characteristics. Estimates are to be evaluated as averages over all the possible combinations
of the other components. Values on the Y-axis represent the reference category. Point estimates in red
represent effects for respondents younger than 57, while those in blue are for those older than that.

20
 Figure 4: Preferences for Twitter profiles, by respondent digital literacy
(a) Full sample (N = 4,194), internet skills

(b) Full sample (N = 4,194), power user

The figure reports combined plots for estimates of the change in probability of selecting the candidate,
based on the reported characteristics. Estimates are to be evaluated as averages over all the possible
combinations of the other components. Values on the Y-axis represent the reference category. Point
estimates broken down by respondent quantile of digital literacy.

21
to have polarizing effects. In the attention check sample (but not the full sample),
we see Republicans penalizing college graduates and Ivy League college graduates
compared to high school graduates; Democrats demonstrate a significant preference
for Ivy League grads. This is potential evidence in favor of the growing cultural
polarization between Democrats and Republicans with respect to the status of higher
education. We suggest future research.

4.1 Alternative Subgroup Analysis

Following Leeper, Hobolt, and Tilley (2020), we replicate our subgroup analysis,
estimating the marginal mean of preference for each attribute rather than the AMCE.
We omit the results broken down by digital literacy (either scale) for space; there’s
nothing going on there.
Figure 6b(a) shows that our hypothesis about the effect of social feedback was
correct—there is a signficant differenece in the marginal mean of the level of so-
cial feedback between Baby Boomers (and older) and younger generations. Even
more encouragingly, for this central mediator of digital media effects, we only find
treatment effect heterogeneity in this attribute, not in any of the other eight.
Figure 6b(b) shows results that are broadly similar to the primary analysis of
partisan heterogeneity, with slightly less crisp results about the way that Republicans
and Democrats respond to politicians based on their education.

5 Conclusions

This paper extends recent work to increase the validity of conjoint experiments in
political science. We argue that the use of a visual conjoint is higher in ecological validity
than is the standard form of conjoint.

22
 Figure 5: Preferences for Twitter profiles, by respondent partisanship
(a) Lucid, full sample (N = 4,194)

(b) Lucid, attention checks (N = 2,570)

The figure reports OLS estimates of the change in probability of voting for candidate, based on the
reported characteristics. Estimates are to be evaluated as averages over all the possible combinations
of the other components. Values on the Y-axis represent the reference category.

23
 Figure 6: Subgroup Analysis with Difference in Marginal Means
(a) By Respondent Generation (attention check, N = 2,570)

(b) By Respondent Partisanship (attention check, N = 2,570)

The figure reports OLS estimates of the change in marginal mean of preferring a candidate, based
on the reported characteristics, relative to a baseline of under 57 years old (top panel) or no party ID
(bottom panel)

24
 We replicated the results of the standard conjoint in Hainmueller, Hopkins, and
Yamamoto (2014a), demonstrating that our approach shares many of the desirable
properties of the now-widely-used experimental design.
Substantively, we demonstrate the additional value of the visual conjoint over
the standard approach in the context of social media profiles. We find that high
levels of social feedback cause respondents to be more favorable—but only among
respondents who are Baby Boomers or older. This result qualifies recent evidence
by Conzo et al. (2021) that social media reactions might influence views only for a
subset of social media users.
Methodologically, we advocate for the use of visual conjoints (specifically, imi-
tating social media profiles or related contexts in which citizens actually confront
information-bundles of democratic importance) to answer a variety of questions of
interest to political scientists.
We provide python code for generating the images that comprise the visual con-
joint at scale at:
https://github.com/avecchiato/Introducing_Visual_Conjoints
We hope to provide an integrated workflow that includes implementation of the
randomization of these images in Qualtrics; at present, this remains the biggest
roadblock.

25
References

Abramson, Scott F, Korhan Koçak, and Asya Magazinnik. 2019. What Do We Learn
About Voter Preferences From Conjoint Experiments? Technical report Working
paper. URL: https://scholar. princeton. edu/sites/default/files . . . .

Aronow, Peter Michael, Joshua Kalla, Lilla Orr, and John Ternovski. 2020. “Evidence
of Rising Rates of Inattentiveness on Lucid in 2020.”.

Bansak, Kirk, Jens Hainmueller, Daniel J Hopkins, and Teppei Yamamoto. 2017.
“Beyond the breaking point? Survey satisficing in conjoint experiments.”.

Bansak, Kirk, Jens Hainmueller, Daniel J Hopkins, Teppei Yamamoto, James N Druck-
man, and Donald P Green. 2021. “Conjoint survey experiments.” Advances in
Experimental Political Science 19.

Conzo, Pierluigi, Juan S Morales, Andrea Gallice, Laura Taylor, and Margaret
Samahita. 2021. “Can Hearts Change Minds? Social Media Endorsements and
Policy Preferences.” UCD CENTRE FOR ECONOMIC RESEARCH .

Coppock, Alexander. 2018. “Generalizing from survey experiments conducted on

mechanical Turk: A replication approach.” Political Science Research and Methods
pp. 1–16.

De la Cuesta, Brandon, Naoki Egami, and Kosuke Imai. 2019. “Improving the external
validity of conjoint analysis: The essential role of profile distribution.” Political
Analysis pp. 1–27.

Dijkstra, Jan, and Harry JP Timmermans. 1998. “Conjoint Analysis and Virtual
Reality–a Review.”.

26
Dominique-Ferreira, Sérgio, Antonio Rial Boubeta, and Jesús Varela Mallou. 2012.
“Minimizing the risks of innovation in bottled water design: An application of
conjoint analysis and focus group.” African Journal of Business Management 6 (31):
9096–9104.

Edwards, Rosalind, and Chamion Caballero. 2008. “What’s in a Name? An Explo-

ration of the Significance of Personal Naming of ‘Mixed’ Children for Parents from
Different Racial, Ethnic and Faith Backgrounds.” The Sociological Review 56: 39–60.
URL: http://journals.sagepub.com/doi/10.1111/j.1467-954X.2008.00776.x

Egami, Naoki, and Kosuke Imai. 2018. “Causal interaction in factorial experiments:
Application to conjoint analysis.” Journal of the American Statistical Association .

Goffman, Erving et al. 1978. The presentation of self in everyday life. Harmondsworth
London.

Guess, Andy, and Kevin Munger. 2020. “Digital Literacy and Online Political Behav-
ior.” Charlottesville: OSF Preprints. Retrieved April 13: 2020.

Hainmueller, Jens, Daniel J. Hopkins, and Teppei Yamamoto. 2014a. “Causal in-
ference in conjoint analysis: Understanding multidimensional choices via stated
preference experiments.” Political Analysis .

Hainmueller, Jens, Daniel J Hopkins, and Teppei Yamamoto. 2014b. “Causal in-
ference in conjoint analysis: Understanding multidimensional choices via stated
preference experiments.” Political Analysis 22 (1): 1–30.

Hargittai, Eszter. 2009. “An update on survey measures of web-oriented digital

literacy.” Social science computer review 27 (1): 130–137.

27
Intille, Stephen, Charles Kukla, and Xiaoyi Ma. 2002. Eliciting user preferences using
image-based experience sampling and reflection. In CHI’02 Extended Abstracts on
Human Factors in Computing Systems. pp. 738–739.

Jenke, Libby, Kirk Bansak, Jens Hainmueller, and Dominik Hangartner. 2019. “Using
Eye-Tracking to Understand Decision-Making in Conjoint Experiments.” Political
Analysis pp. 1–27.

Leeper, Thomas J, Sara B Hobolt, and James Tilley. 2020. “Measuring subgroup
preferences in conjoint experiments.” Political Analysis 28 (2): 207–221.

McClean, Charles, and Yoshikuni Ono. 2020. How Do Voters Evaluate the Age of
Politicians? Technical report.

Messing, Solomon, and Sean J Westwood. 2014. “Selective exposure in the age of
social media: Endorsements trump partisan source affiliation when selecting news
online.” Communication Research 41 (8): 1042–1063.

Munger, Kevin. 2017. “Tweetment effects on the tweeted: Experimentally reducing

racist harassment.” Political Behavior 39 (3): 629–649.

Munger, Kevin. 2019. “All the News That’s Fit to Click: The Economics of Clickbait
Media.” Political Communication pp. 1–22.

Munger, Kevin. 2020. “Don’t@ me: Experimentally reducing partisan incivility on

Twitter.” Journal of Experimental Political Science pp. 1–15.

Munger, Kevin, Ishita Gopal, Jonathan Nagler, and Joshua A Tucker. 2021. “Acces-
sibility and generalizability: Are social media effects moderated by age or digital
literacy?” Research & Politics 8 (2): 20531680211016968.

28
Sundar, S Shyam, and Sampada S Marathe. 2010. “Personalization versus customiza-
tion: The importance of agency, privacy, and power usage.” Human Communication
Research 36 (3): 298–322.

Sylcott, Brian, Seth Orsborn, and Jonathan Cagan. 2016. “The effect of product
representation in visual conjoint analysis.” Journal of Mechanical Design 138 (10).

29
A Questionnaire

30
Digital Home Style

Start of Block: Consent

Q1.1 DESCRIPTION: This research project is being conducted by Kevin Munger from Penn State
University and Alessandro Vecchiato from Stanford University. It is a study to learn more about how
people interact with online politics. Your participation is voluntary. Participation involves completion of
a short survey. TIME INVOLVEMENT: Your participation will take approximately 10 minutes for the
survey. RISKS AND BENEFITS: The risks associated with this study are none. The benefits which may
reasonably be expected to result from this study are none. PAYMENTS: You will receive 1 dollar as
payment for your participation in the questionnaire. PARTICIPANT’S RIGHTS: If you have read this form
and have decided to participate in this project, please understand your participation is voluntary and
you have the right to withdraw your consent or discontinue participation at any time without penalty or
loss of benefits to which you are otherwise entitled. The alternative is not to participate. You have the
right to refuse to answer particular questions. The results of this research study may be presented at
scientific or professional meetings or published in scientific journals. Your individual privacy will be
maintained in all published and written data resulting from the study. Identifiers might be removed
from identifiable private information and, after such removal, the information could be used for future
research studies or distributed to another investigator for future research studies without additional
informed consent from you. CONTACT INFORMATION:Questions: If you have any questions, concerns
or complaints about this research, its procedures, risks and benefits, contact the Protocol Director,
Alessandro Vecchiato, at vecchiato@stanford.edu (please put “Digital Home Style” in the Subject of your
message). Independent Contact: If you are not satisfied with how this study is being conducted, or if
you have any concerns, complaints, or general questions about the research or your rights as a
participant, please contact the Stanford Institutional Review Board (IRB) to speak to someone
independent of the research team at (650)-723-2480 or toll free at 1-866-680-2906, or email at IRB2-
Manager@lists.stanford.edu. You can also write to the Stanford IRB, Stanford University, 1705 El
Camino Real, Palo Alto, CA 94306. Please save or print a copy of this consent form for your records.
Do you agree to participate in this study?

o Yes (1)
o No (2)
Page Break
Q1.2 For our research, careful attention to survey questions is critical! To show that you are paying
attention please select "I have a question."

o I understand (1)
o I do not understand (2)
o I have a question (3)
Page Break
Q1.3 We want to get your opinion on a number of political topics of interest to the US population today.
There are not any right or wrong answers to these questions. But to ensure that you are reading each
question carefully, we need you to select the sixth option out of the following eight options.

o Taxes (1)
o Economy (2)
o Health (3)
o Immigration (4)
o Foreign Affairs (5)
o Climate change (6)
o Religion (7)
o Hard Work (8)

Q2.1 How old are you?

________________________________________________________________

Q2.2 What is your sex?

o Male (1)
o Female (2)
Q2.3 What is your ethnicity?

o White (1)
o Hispanic or Latino (2)
o Black or African American (3)
o Native American or American Indian (4)
o Asian/Pacific Islander (5)
o Other (6)

Q2.4 Generally speaking, do you usually think of yourself as a Democrat, a Republican, an independent?

o Strong Democrat (1)

o Democrat (2)
o Not very strong Democrat (3)
o Independent (4)
o Not very strong Republican (5)
o Republican (6)
o Strong Republican (7)
Q2.5 How often do you use the Internet?

o Pretty much all the time (1)

o Several times a day (2)
o About once a day (3)
o 3 to 6 days a week (4)
o 1 to 2 days a week (5)
o Every few weeks (6)
o Less often (7)

Q2.6 How often do you use Facebook?

o Pretty much all the time (1)

o Several times a day (2)
o About once a day (3)
o 3 to 6 days a week (4)
o 1 to 2 days a week (5)
o Every few weeks (6)
o Less often (7)
o I don't use Facebook (8)
Page Break
Q2.7 How familiar are you with the following computer and Internet-related items? Please choose a
number between 1 and 5 where 1 represents “no understanding” and 5 represents “full understanding”
of the item.
No Full
understanding 2 (2) 3 (3) 4 (4) understanding
(1) (5)

Advanced
search (1) o o o o o
Favorites (2)
o o o o o
Preference
setting (3) o o o o o
PDF (4)
o o o o o
Cache (5)
o o o o o
Wiki (6)
o o o o o
JPG (7)
o o o o o
Blog (8)
o o o o o
Malware (9)
o o o o o
Phishing (10)
o o o o o
Hashtag (11)
o o o o o
RSS (12)
o o o o o
Selfie (13)
o o o o o
Tagging (14)
o o o o o
Page Break
Q2.8 Please indicate your agreement with the following statements on a scale of -4 = Strongly Disagree
to 4 = Strongly Agree
 -4 4
(Strongly (Strongly
-3 (2) -2 (3) -1 (4) 0 (5) 1 (6) 2 (7) 3 (8)
Disagree) Agree)
(1) (9)

I prefer to
ask friends
how to use
any new
technological
gadget o o o o o o o o o
instead of
trying to
figure it out
myself (1)

I feel like
information
technology is
a part of my
o o o o o o o o o
daily life (2)

Using
information
technology
makes it o o o o o o o o o
easier to do
my work (3)

I often have
trouble
finding
things that
I've saved on
o o o o o o o o o
my
computer (4)
 I have
problems
with viruses
and malware
on my
o o o o o o o o o
computer.
(5)

I rely on
family
members to
introduce
me to new
o o o o o o o o o
technology.
(6)

I have
professionals
(such as the
Geek Squad)
or family
members
take a look
at my
o o o o o o o o o
computer
when
something
isn’t
working. (7)

Q2.9 How many words are in this sentence?

________________________________________________________________
Q2.10 How often do you read news stories online?

o Pretty much all the time (1)

o Several times a day (2)
o About once a day (3)
o 3 to 6 days a week (4)
o 1 to 2 days a week (5)
o Every few weeks (6)
o Less often (7)

Q2.11 How often do you read news stories offline (in the newspaper, printed news magazines)?

o Pretty much all the time (1)

o Several times a day (2)
o About once a day (3)
o 3 to 6 days a week (4)
o 1 to 2 days a week (5)
o Every few weeks (6)
o Less often (7)
End of Block: Descriptives

Start of Block: Intro To Experiment

Q66 In this section, we will show you a series of Twitter profiles of hypothetical Members of the US
Congress. These profiles (and the people they represent) are not real, but are constructed from
composites of real people and profiles.
For each pair of profiles, please take your time and read each of the profiles carefully, then tell us which
politician you'd prefer to be your representative in Congress.

End of Block: Intro To Experiment

Start of Block: Republican - Alabama

Display This Question:
If GeoIP Location Region = AL

Q68 Which of these politicians would you rather have as your Representative in Congress?

o Image:Bl woman boomer high national (31)

o Image:Bl woman boomer high local (32)