HCI Reading Assignment 2

1. Summary

1.1. Motivation
This research paper focuses on the influence of demand characteristics in HCI research, particularly in studies
involving underprivileged or socially disadvantaged groups. Demand characteristics refer to a phenomenon
observed in research studies where the study participants change their behavior to align with what they believe
the goals of the interviewer are. This phenomenon can lead to large inaccuracies in the reported results of HCI
studies. This research paper attempts to quantify the significance of the bias from interviewer demand
characteristics. It also investigates how the demand characteristic varies with the difference in the socioeconomic
status or class of the investigators and participants.

1.2. Contribution
This research study attempts to bring the concept of demand characteristics to the attention of existing HCI and
HCI4D researchers. It further tries to quantify the response bias associated with demand characteristics in two
specific contexts and prove the accuracy of the following hypotheses:

• The participants in a study tend to favor the response that they believe is favored by the interviewer.

• This bias towards the response favored by the interviewer is even larger if the interviewer is a foreign
individual with a translator. (This would show that the bias tends to increase with the perceived gap in the
social status of the interviewer and the participant.)

Overall, the study tries to sensitize researchers about such effects (which are particularly prevalent when dealing
with socioeconomically disadvantaged populations), so that studies in the future consider the role of demand
characteristics before reporting their results.

1.3. Methodology
The field study was conducted using 450 total participants from 2 separate populations – IISc graduate
students and auto drivers in Bangalore over a time period of 5 weeks. The sample size for each experiment was
50.

In the first experiment, the interviewer associated herself with one of the video players being tested. Then the
participants were shown identical videos on 2 video players on 2 phones (with 1 video player per phone). The
participant was then asked about which video player they preferred or whether they thought both had the
same quality. This experiment was conducted with both populations and with both a local interviewer and a
foreign interviewer with a local translator. Here, it was found there was no meaningful difference in the
responses of graduate students for both interviewer cases so for the second experiment, the graduate
students were just interviewed by the foreign interviewer with a local translator.

In the second experiment, the experiment was tried with both the interviewer associating with 1 video and not
associating with either video. In this experiment, one of the videos was of a noticeably lower quality than the
other and the study participants were again asked about which video they preferred or whether they thought
both had the same quality. The experiment was conducted for auto drivers with both interview cases and it
was conducted for graduate students with just the foreign interviewer.

The results of both of these experiments were carefully noted and checked for statistical significance.

1.4. Conclusion
For the first experiment where two identical videos were shown, this paper measured response bias as the ratio of
the number of individuals who chose the video associated with the interviewer to the number of individuals who
chose the other video. This ratio was on average equal to 2.5 across all 4 experiment conditions, showing that
there is indeed a strong bias of a study’s participants towards the response associated with the interviewer. In
addition, it was found that the largest bias of 5 occurred when the foreign researcher interviewed the auto drivers
and the smallest bias of 1.7 occurred when the local researcher interviewed graduate students with the other 2
cases having intermediate bias values.

For the second experiment where one of the videos had noticeably lower quality, if the interviewer did not
associate themselves with either of the videos, then both populations tended to favor the high-quality video in
their response. Here, we can see how the lack of interviewer-related bias led to more accurate results. When the
interviewer did associate themselves with one of the videos, a lower percentage of the graduate population chose
the better quality video for their response as compared to before. Similarly, when the interviewer did associate
themselves with one of the videos and the interviewer was a local, for the auto driver population, an even lower
percentage chose the better quality video for their response as compared to before. Overall, however the better
quality video still constituted a majority of responses. However, when a foreign interviewer associated themselves
with one of the videos, then the percentage of auto drivers who chose the lower quality video was actually higher
than the percentage that chose the higher quality video.

Both these experiments show that demand characteristic related response bias is a very real problem that affects
the accuracy of research surveys. This response bias could be so high that an objectively incorrect response could
be chosen by a study participant if they believe that that’s the result that the interviewer wants. In addition, in
general, the response bias associated with demand characteristics increases with the perceived gap in the
socioeconomic status of the interviewer and the participant.

2. Critique

2.1. Lack of Diversity in Study Participants

This research study was conducted with only 450 participants across 2 populations – male auto drivers and male
IISc graduate students in one city - Bangalore. The results of this study that were obtained for these 2 specific
populations may not necessarily be generalizable to the entire population of India or people in general. Further, if
this study were conducted for a more representative sample of the population, then its results would be more
accurate to the entire population.

2.2. Disregard of Other Sources of Bias Besides Demand Characteristics

This research study was conducted to identify the quantitative effects of response bias related to demand
characteristics in research surveys. However, in the process of conducting these 2 experiments, other sources of
bias that may have affected the collected responses were ignored. For example, the gender of the participants and
interviewer may had a substantial impact on the responses. In addition, other factors like caste, personal interest
in technology etc. may have also had meaningful impacts on the result of their experiment but data related to
them was not collected.

3. Synthesis

3.1. Analysis of How Response Bias Associated with Demand Characteristics Change with the
Characteristics of the Interviewer and Participant
In this research study, it was observed that response bias was minimum when local interviewers were interviewing
graduate students, and that the response bias was maximum when foreign interviewers were interviewing auto
drivers. This showed that in general, response bias associated with demand characteristics increases with
perceived gap in the socioeconomic status of the interviewer and the participant. This could be investigated
further by having the interviewer and participant be of different castes or income levels etc. The objective here
would be to find the factors that have the most and least influence on demand characteristic associated response
bias so that future surveys could minimize such bias.
3.2. Analysis of How to Conduct Surveys while considering the Effects of Demand
Characteristics
Response bias associated with demand characteristics will always be a factor in the responses to surveys. This is
because survey participants often share the hope with the experimenter that the study will be successful.
Frequently, survey participants either consciously or subconsciously take actions in a study to provide the
experimenter with what they believe to be the “right” results. In the absence of an obvious experimental outcome,
participants tend to guess the intended outcome and change their behavior accordingly.

Thus since demand characteristics can’t be eliminated, it would be a good idea to research alternative ways of
using surveys to obtain information that take into account the effect of demand characteristics before reporting a
result. This can include techniques ranging from anonymous surveys to using techniques like including filler
questions in survey and using a mix of question types to mask the goal of the survey. These techniques should be
studied and compared on the basis of how effective they are at reducing response bias.