Proceedings of the 2017 Industrial and Systems Engineering Conference

K. Coperich, E. Cudney, H. Nembhard, eds.

An Empirical Study of Crowdsourcing Services

using Repertory Grid

Abstract ID: 2969

Tasnia Reza, Jong Youl Lee, Kyoung-Yun Kim

Department of Industrial and Systems Engineering
Wayne State University
Detroit, MI USA

Abstract
A new form of distributed labor, namely, crowdsourcing is used to solve the real-world business challenges in
different fields of services by using people crowds. Crowdsourcing gathers and generates concepts by adopting
principles of distributed computing, open networks, and parallel processing. However, currently, there are limited
formal methods or guidelines to analyze and develop crowdsourcing services. As a result, there is no clear
understanding of different characteristics of various crowdsourcing services. The aim of this paper is to extract the
critical factors as guidelines for successful crowdsourcing service development. In this paper, eight crowdsourcing
services are analyzed using Repertory Grid, in order to identify the critical characteristics of the crowdsourcing
services. In this study, 25 personal constructs are developed as key factors related to the successfulness of the
crowdsourcing services. The services and constructs are analyzed with the aid of three correlation measures (e.g.,
Pearson, Spearman, and Kendal measures). We observe constructs with highly positive correlations and constructs
with highly negative correlations with the ‘successfulness’ construct. As concluding remarks, we discuss the
analysis results and future research directions.

Keywords
Crowdsourcing, repertory grid, empirical study, personal construct, correlation analysis.

1. Introduction
Crowdsourcing is an emerging paradigm of concept gathering and generation. The power of crowd is being utilized
by adopting principles of distributed computing, open networks, and parallel processing [1]. Although
crowdsourcing is a single word, it has different characteristics for different services. A clear understanding of these
differences like, how the process for different crowdsourcing services varies based on problem criteria, selection
process, target audience, market strategy, motivation for crowds to participate, etc., is limited [2]. Some reason of
the limited understanding is the lack of a formal method to analyze crowdsourcing services and a clear guideline for
crowdsourcing service development. To build such guidelines, a good understanding of the sheer number of
emerging crowdsourcing services is a must. Thus, in this paper, Repertory Grid is used to identify the critical factors
of successful crowdsourcing services by analyzing eight crowdsourcing services with different service process
characteristics. The objective of this study is to extract the critical factors as guidelines for successful crowdsourcing
service development. Next sections will cover the concepts of crowdsourcing and repertory grid.

2. Background: Crowdsourcing and Repertory Grid

2.1 Crowdsourcing
Crowdsourcing is a web-based business model that exploits the creative solutions of a group of individuals active in
a distributed network through an open call for proposals. J. Howe and M. Robinson first introduced the term
“Crowdsourcing" in the June 2006 issue of Wired magazine [3]. According to them: “Crowdsourcing is the act of

1556
 Reza, Lee, and Kim
taking a job traditionally performed by a designated agent (usually an employee) and outsourcing it to an undefined,
generally large group of people in the form of an open call” [3][4].

Further clarification from Howe indicates that, “it’s only crowdsourcing once a company takes that design,
fabricates [it] in mass quantity and sell[s] it” [3]. In other words, a problem is posted online, solution is offered by a
broad number of individuals, some form of prize is awarded to the winning ideas, and the winning idea is mass
produced by the company for its own benefit. A clear understanding of the crowdsourcing process is necessary to
conduct a successful business by using the power of distributed crowds online. It is actually more meaningful to
examine some of the most successful and profitable crowdsourcing ventures in a wide range of industries to better
understand the process of crowdsourcing.

Focus on solving organizational problems and performing organizational tasks is presented in a model in Brabham’s
work [1]. Feasibilities for the model, its likelihood to harness a crowd of innovators, and its potential for use beyond
for profit sectors is also discussed in [1]. Geiger et al. [2] view crowdsourcing as a process, and categorize it by the
nature of the process. Whitla [3] examines how firms are using crowdsourcing for market related task completion.
Vukovic and Bartolini discuss the role of crowdsourcing for the context of the enterprise [5]. Schenk et al. [6]
present the typology of crowdsourcing practices and discuss the notion in the perspective of management science.
Kim et al. [7] address the crowdsourced design team formation problem with the actor network theory approach.

2.2 Repertory Grid

Recent studies have established the importance of understanding the organizational cognitions [8] and process of
crowdsourcing ventures. A theory and method suitable to the study of cognition technique is Kelly's (1955) Personal
Construct Theory and its cognitive mapping tool is known as the repertory grid. This technique was originally
developed for use in clinical psychology, but it has been applied in many other fields [9]. In repertory grid, an
interview is carried out in a highly-structured manner by using the interviewee's own language and setting out their
responses in the form of a grid. The repertory grid study includes four main components: topic selection, element
development, construct development, and rating. Topic covers what the interview is about. Elements are examples
that illustrate the topic and define the scope of the study area. Elements are compared with each other to produce a
series of statements describing what the interviewee thinks about the research topic. These statements are called
personal constructs or constructs. They are the hypotheses constantly being tested and revised as per the
interviewee’s adaption towards the changes. These statements are eventually formed into bi-polar unit of analysis,
contrasting similarities and differences (e.g., hot or cold). After having main constructs and elements in place, a grid
is created with elements on top and the constructs down the side. The interviewee rates each element against each
construct according to a rating scale, usually of 1 to 7. It evaluates the elements in terms of each bipolar construct
[10].

A big advantage of using the repertory grid technique is that it allows interviewees to articulate their experience in
the way they see the problem domain, per their own personal constructs. So, this technique offers the potential to
significantly enhance the understanding of crowdsourcing processes and identify the most significant factors to run
the process successfully. The diagnostic qualities of the repertory grid and its mapping outcomes can also be used
for the practical intervention at the individual and organizational levels. So, the use of repertory grid for this study is
much appropriate as the aim, which is to understand and identify the factors related to the successfulness of the
crowdsourcing services.

3. Repertory Grid Construction

This section introduces eight crowdsourcing services as elements, 25 constructs, the rating for the grid, and the grid
analysis for the eight crowdsourcing services (elements).

3.1 Element Selection

Eight crowdsourcing services (Table 1) are selected as elements (i.e., MyStarbucks Idea, InnoCentive, Lays, Quirky,
Threadless, Lego Ideas, Openstreet map, and Kickstarter). We evaluate various other services and selected these
eight elements, which are very different in terms of their service processes; each service has unique process
characteristics. For the process analysis, the services are categorized based on the type of crowdsourcing tasks,
target audience, market strategy, type of labor performed, selection process of final solution, and motivation for the
crowds to participate. This process analysis will be reported in a separated article.

1557
 Reza, Lee, and Kim
Table 1: List of crowdsourcing services

No Crowdsourcing service Website

1 MyStarbucks Idea http://mystarbucksidea.force.com/apex/ideahome
2 InnoCentive https://www.innocentive.com
3 Lays http://www.fritolay.com
4 Quirky https://www.quirky.com
5 Threadless https://www.threadless.com
6 Lego Ideas https://ideas.lego.com
7 Openstreet map https://www.openstreetmap.org
8 Kickstarter https://www.kickstarter.com

3.2 Constructs with Rating Grid

The repertory grid technique is based on a way of exploring the format and content of such implicit theories in its
multitude of forms. This overall theory is linked with other smaller theories and the whole system is called a
personal construct system [11]. To establish the most significant constructs, five individuals are interviewed about
eight crowdsourcing services, which are referred as elements for this study. Their personal constructs are recorded
and eventually 25 most significant ones surfaced as permanent constructs for this research. Data collection is
conducted in two steps. First, constructs are elicited by using all elements at least once in the elicitation process. The
interviewees are asked to list all the similar as well as different characteristics of each element. These characteristics
are converted as constructs for the repertory grid analysis. Three sets of the assorted bipolar construct lists are
collected from two teams of two members and one individual. These personal constructs are summarized by getting
rid of the duplicate and irrelevant ones compared to the scope of study area. In this way, every constructs are related
to all eight elements of this study. Then, the constructs are grouped in clusters with similar kinds to better
understand their relationship with each other.

For the second stage of data collection, the five interviewees are same but they participated as individuals this time.
They are asked to rate the same eight elements on a scale of 1 to 7 using 25 fixed elicited constructs (Table 2). 7 is
used for lower and 1 is for higher attributes. So, this time five sets of fixed bipolar construct lists with rankings are
collected. Medians were taken from five ranking data sets to get the final ranks. The following grid is created with
elements on top and the constructs down both side. The five interviewee rates eight elements against each construct
according to a rating scale, 1 to 7. It helps to evaluate the elements in terms of each bipolar construct by creating a
linkage between them. Table 2 shows the list of constructs and their ratings. The detailed analysis results are
discussed in the following section.

Table 2: List of 25 constructs with the rating grid

MyStarbucks Idea

Openstreet map
InnoCentive

Kickstarter
Threadless

Lego Ideas
Quirky
Lays

Construct group ID 1 7 Description

Good communication Bad communication with Level of communication with

1 2 3 1 3 2 4 3 3
with crowd crowd the crowdsourcing participant
Communication
How clearly the instructions are
More Process Less Process instruction
2 4 2 2 2 1 2 4 3 provided for participating
instruction for crowd for crowd
crowds
How easy it is to navigate the
website and participate using the
3 More user friendly 6 2 2 4 2 2 4 3 Less user friendly website Level of filtering by the
authority and intermediate
Control
supervision
Level of filtering by the
High manager’s power Low manager’s power on
4 5 1 2 1 2 2 5 6 authority and intermediate
on selection process selection process
supervision
5 Innovative Idea 4 1 2 5 1 2 6 1 Conservative Idea The creativity level of any idea
Idea
6 High flexibility in idea 3 2 2 5 2 3 4 2 Low flexibility in idea Adaptability of the idea

1558
 Reza, Lee, and Kim
The problem-solving style &
7 Experience based 2 5 1 5 2 4 2 3 Procedure based whether past experiences have
Product an effect or not
The uniformity of solutions to
8 Heterogeneous solution 1 1 3 4 2 3 4 4 Homogeneous solution
different cases
Project data availability on
High accessibility to
9 6 2 1 2 1 2 2 5 Low accessibility to data website – crowds can see other
data
project data
Data
The amount of information/
10 Satisfactory data 6 7 2 3 4 3 3 5 Limited data knowledge about the problem
available for decision-making
The influence of time on the
Time 11 More time dependency 2 1 2 2 4 2 2 3 Less time independency
problem or solution

More diverse skilled Less diverse skilled

Skill 12 5 5 2 3 4 3 6 4 Level of skill of the participants
participant participant

More crowd
Few crowd participation
13 participation (service 5 3 1 4 3 2 2 1
(Service website)
website)
More crowd
Participation Few crowd participation
14 participation (other 7 2 3 3 2 2 5 5
(other websites)
websites)
More organization Few organization member
15 2 3 3 5 6 4 6 6
member participation participation
Whether there is enough reward
High Motivation to Less Motivation to
16 3 1 6 5 3 4 5 2 for the crowd to feel motivated
contribute contribute
to participate
More % of Royalty on Less % of Royalty on The royalty given to that portion
Motivation 17 3 7 7 4 2 3 7 5
sales sales of the crowd that gave solution
The recognition given to that
18 More Social recognition 5 2 1 5 4 2 5 3 Less Social recognition portion of the crowd that gave
solution
Multiple competitors in Competitors of the
19 6 3 7 3 5 7 3 4 Monopoly market
market crowdsourcing venture
The cost incurred when making
20 High failure risk 3 2 6 3 3 4 4 4 Low failure risk
the wrong decision
The number of past cases that
21 Many cases 4 4 3 4 5 5 3 2 Few cases
could be consulted
Multiple Homogeneous
Other 22 5 4 4 2 3 4 4 2 Service type in relevant field
Products/Services Product/Service
23 Multiple solution owner 3 5 5 2 5 4 6 4 Single solution owner
High Project Success
24 3 6 4 4 4 4 2 3 Low Project Success Rate How many projects succeed?
Rate
Whether the venture itself is
25 Successful venture 3 2 2 6 2 2 3 2 Unsuccessful venture
successful or not as a whole

4. Analysis Results

In this section, the services and constructs are analyzed with the aid of three correlation measures (e.g., Pearson,
Spearman, and Kendal measures) and the findings are described. Correlation is an assessment of the strength of
association between any two variables. The numerical range of the correlation coefficient is between -1 to +1. The
value being close to ± 1 indicates a strong correlation. The closer it is to zero, the weaker the correlation is.

To find which factors make the eight crowdsourcing services successful or unsuccessful, each construct is used to
compare with the rest of 24 constructs. All constructs are made and ranked by the subjective criteria. The focus for
this study is to identify the relationship between each construct with “successful venture and unsuccessful venture”
constructs. Figure 1 shows the result of each Pearson correlation coefficient among 25 constructs.

The standard value is set for the positive relationship as the correlation coefficient of + 0.6 or more. If the value of
correlation coefficient between ‘successful crowdsourcing venture’ construct (Construct No. 25) and the other
constructs (Constructs No. 1 to 24) is greater than or equal to +0.6, it could be concluded that the construct is a
positive factor to the success of the crowdsourcing ventures. Construct IDs and descriptions are displayed in Table 2.
Likewise, the standard value is set for the negative relationship standard as -0.6 or less. If the value of correlation
coefficient between the successfulness construct (Construct No. 25) and the other constructs (Construct No. 1 to 24)
is less than or equal to -0.6, it could be concluded that the construct is a negative factor to the success of the
crowdsourcing ventures.

1559
 Reza, Lee, and Kim

Figure 1: Correlation analysis: Pearson

In this analysis, the construct of ‘high flexibility in idea’ turned out to be the highest positive correlation (0.89) with
crowdsourcing venture’s success, followed by the construct of ‘how much the idea innovative’ (0.70) and the
construct of ‘social recognition of the venture’ (0.63). On the other hand, ‘the number of solution owner’ is revealed
to high negative correlation with crowdsourcing venture’s success.

Table 3: List of critical factors for successful crowdsourcing ventures

The relationship with the ‘success of crowdsourcing

Standard of correlation coefficient
ventures’
 The level of flexibility in idea the venture has
Positive factors  The level of innovative idea the venture has
 The level of social recognition the venture has
Negative factor  The number of solution owner

We also conduct two additional correlation analyses with Spearman and Kendal measures to confirm the consistency
of the correlations as shown in Table 4. The bold-highlighted numbers in Table 4 indicate significant (above 0.6)
correlation values. Due to the space limit, the constructs that receive significant values are included in Table 4.
Spearman correlation is often used to assess how well the relationship between two variables can be measured. In
contrast, the Kendall coefficient finds how much similarity between two sets of ranks in the situation of a same set
of objects [12]. According to [13], the Pearson correlation is the most efficient, but the Spearman and Kendall shows
a good compromise between robustness and efficiency. As reported in Table 4, both analyses in the Spearman and
Kendall correlation, an additional construct, ‘the level of user friendliness the venture has’ is included in the positive
factors of successful crowdsourcing services. Another noticeable difference is that the Spearman correlation
measure has the highest degree of the correlation in three positive factors among the other two correlation measures.
The Kendall correlation values exceed Pearson correlation values in No. 5 and No. 18. However, Spearman and
Kendall correlation measures do not show significant negative correlation of all constructs.

Table 4: Significant correlations of measures with Construct No. 25

Correlation Measure and

3 5 6 18 23
Construct ID
Pearson 0.51 0.70 0.89 0.63 -0.68
Spearman 0.86 0.85 0.88 0.85 -0.40
Kendal 0.74 0.74 0.85 0.74 -0.35

1560
 Reza, Lee, and Kim

5. Concluding Remarks
This study reported a repertory grid analysis results and elements and constructs building processes. The extracted
constructs can be used as guidelines for crowdsourcing service development. However, this research is based on
subjective data and rating. According to the Pearson correlation analysis, ‘the level of flexibility in idea the venture
has,’ ‘the level of innovative idea the venture has,’ ‘the level of social recognition the venture has’ are indicated as
highly positive correlated factors with the successfulness of crowdsourcing service. ‘The number of solution owner’
is indicated as a highly negative factor, even though the other two correlation measures do not indicate the
significant correlation. For future study, more elements will be included for the analysis. In that research, a research
focus can be on changes from the alteration of sample sizes. In that case, not only unsuccessful elements but also the
elements from non-profit organization can be added for more balanced research. Also, we will continue to add
participants to the repertory grid analysis. Also, the criterion of the successfulness needs to be detailed to minimize
the difference of subjective and biased notion of successfulness. One individual might think success of the service
should be evaluated in market value, the status in its business domain, social reputation, and so on. However, this
limitation is supposed to exist in a subjective evaluation method-based research.

References
1. Brabham DC., 2008; “Crowdsourcing as a model for problem solving: an introduction and cases,”
Convergence: The International Journal of Research into New Media Technologies, 14(1): 75–90.
2. Geiger, D., Seedorf, S., Schulze, T., Nickerson, R.C. and Schader, M., 2011, August, “Managing the
Crowd: Towards a Taxonomy of Crowdsourcing Processes,” In AMCIS
3. Whitla, P., 2009, “Crowdsourcing and its application in marketing activities.” Contemporary Management
Research, 5(1).
4. Howe, J., 2008, “Crowdsourcing: How the power of the crowd is driving the future of business,” Random
House.
5. Vukovic M, Bartolini C., 2010, “Towards a research agenda for enterprise crowdsourcing,” In: Tiziana M,
Bernhard S (eds) Leveraging applications of formal methods, verification, and validation. Lecture Notes in
Computer Science 6415. Berlin/Heidelberg: Springer, pp. 425–434.
6. Schenk, E, Guittard C., 2009, “Crowdsourcing: what can be outsourced to the crowd, and why?” Technical
Report. Available at: http:// halshs.archives-ouvertes.fr/halshs-00439256/ (January 2017).
7. Kim, J., Kim, K.-Y. and Kwon, O., 2016. “Actor Network Theory-based Modeling for Crowdsourced
Design Team Formation,” Journal of Integrated Design and Process Science, 19(4), 2016, 1-25.
8. Tan, Felix B., and M. Gordon Hunter., 2002, “The repertory grid technique: A method for the study of
cognition in information systems,” MIS Quarterly: 39-57.
9. Stewart, V., Stewart, A. and Fonda, N., 1981, “Business applications of repertory grid,” McGraw-Hill
Companies.
10. Easterby-Smith, Mark., 1980, "The design, analysis and interpretation of repertory grids," International
Journal of Man-Machine Studies 13.1: 3-24.
11. Embacher, J. and Buttle, F., 1989, “A repertory grid analysis of Austria's image as a summer vacation
destination,” Journal of Travel Research, 27(3), 3-7.
12. Abdi, H., 2007, “The Kendall rank correlation coefficient,” Encyclopedia of Measurement and Statistics.
Sage, Thousand Oaks, CA, 508-510.
13. Croux, C. and Dehon, C., 2010, “Influence functions of the Spearman and Kendall correlation measures.
Statistical methods & applications,” 19(4), 497-515.

1561
Reproduced with permission of copyright owner. Further reproduction
prohibited without permission.