Received November 8, 2019, accepted November 20, 2019, date of publication November 25, 2019,

date of current version December 9, 2019.

Digital Object Identifier 10.1109/ACCESS.2019.2955677

Collective Intelligence: A Taxonomy and Survey

FEIJUAN HE 1 , (Member, IEEE), YUDAI PAN 2, QIKA LIN 2, XIANGLIN MIAO 1,

AND ZHOUGUO CHEN 3

1 Department of Computer Science, Xi’an Jiaotong University City College, Xi’an 710018, China
2 School of Computer Science and Technology, Xi’an Jiaotong University, Xi’an 710049, China
3 The 30th Research Institute of China Electronics Technology Group Corporation, Chengdu 610041, China

Corresponding author: Qika Lin (qikalin@foxmail.com)

This work was supported in part by the National Key Research and Development Program of China under Grant 2016YFE0206700, and in
part by the National Natural Science Foundation of China under Grant 61877050.

ABSTRACT Collective intelligence (CI) refers to the intelligence that emerges at the macro-level of a
collection and transcends that of the individuals. CI is a continuously popular research topic that is studied
by researchers in different areas, such as sociology, economics, biology, and artificial intelligence. In this
survey, we summarize the works of CI in various fields. First, according to the existence of interactions
between individuals and the feedback mechanism in the aggregation process, we establish CI taxonomy
that includes three paradigms: isolation, collaboration and feedback. We then conduct statistical literature
analysis to explain the differences among three paradigms and their development in recent years. Second,
we elaborate the types of CI under each paradigm and discuss the generation mechanism or theoretical
basis of the different types of CI. Third, we describe certain CI-related applications in 2019, which can be
appropriately categorized by our proposed taxonomy. Finally, we summarize the future research directions
of CI under each paradigm. We hope that this survey helps researchers understand the current conditions of
CI and clears the directions of future research.

INDEX TERMS Collective intelligence, paradigm, synergetics, emergence, survey.

I. INTRODUCTION
Collective intelligence is not a new area of research, and
related research can be traced back to early in the last cen-
tury. To date, collective intelligence is a popular research
topic, attracting researchers from different areas, such as
social science, economics, biology, and artificial intelligence.
Although there is no unified definition of collective intel-
ligence, most researchers agree that collective intelligence
surpasses individual intelligence. Therefore, this survey uses
the definition given by Michelucci and Dickinson [1], that
is, collective intelligence (CI) refers to the intelligence that
emerges at the macro-level of a collection and transcends that
of the individuals.
There are some other terms that denote CI. When individ-
uals are social animals, such as bees, ants and birds, the term
Swarm Intelligence is used to denote CI; otherwise, when the
individuals are humans, the term Wisdom of the Crowd is
used for CI. In this survey, we uniformly use CI to represent
these terms.
There are certain relationships between CI and crowd-
sourcing. Crowdsourcing is a paradigm for using the power
The associate editor coordinating the review of this manuscript and
approving it for publication was Moayad Aloqaily . criteria of CI.
of crowds of people to facilitate large scale tasks that are
costly or time consuming with traditional methods [2]. From
the definition, the targets of CI and crowdsourcing are signif-
icantly different, so we do not discuss studies on crowdsourc-
ing in this survey.
Thus far, we have found three surveys about CI [3]–[5].
Yu et al. [3] presented a literature review on CI in human
society from 1907 to 2017, focusing on the influential works
of researchers. The authors discussed how to measure and
model CI as well as the typical applications and future
directions of CI. Salminen [4] summarized the studies on
CI of human society at three levels. Micro-level research
focuses on the factors enabling CI. Macro-level researchers
regard CI as a statistical or probabilistic phenomenon of
individual output results. Emergence-level studies focus on
how CI emerges at the macro-level from interactions of
individuals at the micro-level. Krause et al. [5] proposed an
overarching definition of CI that covers CI in both human
populations and social animals: individuals acquire infor-
mation independently or partially independently. Through
social interactions, the individuals provide a solution to a
cognitive problem that could not be solved by isolated indi-
viduals. The authors further summarized some important

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see http://creativecommons.org/licenses/by/4.0/
VOLUME 7, 2019 170213

FIGURE 1. Three paradigms of CI.
TABLE 1. Symbols and notations.
Distinguished from previous survey papers, this survey
establishes a CI taxonomy of three paradigms and reviews
the existing studies. The major contributions are from three
aspects:
1) On the basis of whether there are interactions between
individuals and the feedback mechanism in the aggregation
process, we establish a CI taxonomy, including three intelli-
gence paradigms: isolation, collaboration and feedback, from
simple to complicated. We explain their differences and con-
duct certain statistical analysis of literatures.
2) We elaborate the categories of CI under each paradigm
and the corresponding typical studies, and discuss the gen-
eration mechanisms and theoretical bases of different types
of CI.
3) We summarize the challenges and future research direc-
tions of CI under each paradigm.
The survey is organized as follows: The Section II presents
an overview of the three paradigms. Section III, IV and V
give typical research works and explain the CI mechanisms
under three paradigms. Section VI describes certain recent
CI-related applications. Section VII summarizes the possi-
ble future directions of CI. Finally, the Section VIII gives
conclusions.
II. OVERVIEW OF COLLECTIVE INTELLIGENCE
In this section, we introduce an overview of CI tax-
onomy and conduct a statistical literature analysis. The
170214
F. He et al.: Collective Intelligence: A Taxonomy and Survey
general notations used in this survey are shown in Table 1.
The detailed explanations are given in the following
sections.
A. TAXONOMY OF COLLECTIVE INTELLIGENCE
Generally, CI can be regarded as an aggregation of individual
behavior results. According to the existence of interactions
between individuals and the feedback mechanism in the
aggregation process, CI can be divided into three paradigms
including isolation, collaboration and feedback, as Figure 1
shows.
Isolation. The CI of the isolation paradigm works without
interactions and feedback. The individual ai (i ∈ [1, . . . , n])
makes autonomous decisions and produces the behavior
results xi . There are no interactions between the individuals,
so the results xi and xj do not influence each other (xi 6= xj ).
At the collective level, the aggregation method f is used
to process X = (x1 , x2 , . . . , xi , . . . , xn ) and generate f (X ),
where f (X ) does not influence the subsequent individual
behaviors.
The CI of the isolation paradigm mainly appears in human
society and is usually used for estimation, prediction and
association mining tasks. The theoretical basis of this kind
of CI is a set of mathematical theorems, such as Diver-
sity Prediction Theorem [6] and Condorcet’s Jury Theorem
[7], or data mining algorithms, for example, association rule
learning.
VOLUME 7, 2019
F. He et al.: Collective Intelligence: A Taxonomy and Survey

FIGURE 2. Word clouds of three paradigms in related abstracts of papers.

Collaboration. The collaboration paradigm is distin-

guished by working with interactions and no feedback.
Direct interactions or indirect interactions by modifying envi-
ronmental variables between individuals produces behav-
ior result X = (x1 , x2 , . . . , xi , . . . , xn ) and environment E.
On the collective level, the aggregation method f is used to
process X and E, and generate f (X , E), where f (X , E) does
not influence the subsequent individual behaviors.
The CI of the collaboration paradigm mainly exists in
human society and is used to solve complicated tasks that
might be hard for individuals to accomplish. This CI could
be considered a designed serendipity [8]. The group is the
best combination of individuals with complementary abilities
oriented to specific tasks.
Feedback. One of the characteristics of the feedback
paradigm is the existence of interactions and feedback. Direct
interactions between individuals or indirect interactions by
modifying environmental variables produces behavior results FIGURE 3. Proportion of papers under the three paradigms over time.

Xt = (x1,t , x2,t , . . . , xi,t , . . . , xn,t ) and environment Et at

time t. At the collective level, f (Xt , Et ) affects Xt+1 and Et+1 ,
and the CI emerges through a self-organization mechanism.
The CI of the feedback paradigm appears in both human
society and social animals and is mainly used for solving
optimization problems and evolutionary analysis of complex
systems. This kind of CI could be described and explained by
synergetic approaches [9]. Synergetics is an interdisciplinary
research area that is used to illustrate how subsystems in open
complex systems form structures of space, time, or function
through self-organization [10].
The three paradigms mentioned above show that the gen-
eration mechanisms and theoretical bases of CI are quite
different, which indicates that there might not exist a unified
framework to explain various CI phenomena.
B. LITERATURE ANALYSIS
In this survey, we collect 62 papers that cover all three
paradigms by searching for the keywords collective intel-
ligence, swarm intelligence and wisdom of the crowd on
Google Scholar. From the perspective of statistical literature
analysis, we give an intuitive introduction of these studies,
which are shown in Figure 2 and Figure 3.
Figure 2 shows three word clouds for the abstracts of
papers on the three different paradigms. A word cloud is
based on the occurrence frequency of words or word terms
in multiple documents, which are indicated by the position
and size of the corresponding text in the figure. We use the
abstract of each paper as an object document for statistics,
which can highly reflect the concerns of each paper. Accord-
ing to the construction method mentioned in [11], [12], all the
possible word terms are first extracted and the corresponding
frequencies are determined. Then, each word is assigned
a position by an adaptive algorithm. Specifically, the sizes
of the terms depend on their frequencies. Word terms that
are large in size and close to the center are of significance.
In this way, the research of each paradigm can be represented
visually by the corresponding word cloud.
From Figure 2, we can see certain obvious common words
in all the word clouds, such as forecast, accuracy, indi-
vidual, and group. This result reflects the commonality of

VOLUME 7, 2019 170215


TABLE 2. Some representative studies on isolation paradigm.
the three paradigms, that is, they all try to achieve CI by
analyzing or forecasting using the information of individ-
ual or group behaviors. In Figure 2.1, the words individual,
forecast and estimate account for a large part of the cloud.
This result indicates that the isolation paradigm focuses on
some forecasting or estimating tasks of individuals. Corre-
spondingly, in Figure 2.2, the proportion of the words group,
social influence, collective, and team greatly increases. This
result shows that the collaboration paradigm emphasizes the
interactions between groups. In Figure 2.3 of the feedback
paradigm, the frequency of the words algorithm and opti-
mization increases. Combined with the corresponding stud-
ies, it indicates that the feedback paradigm helps to optimize
the solutions of estimates or predictions. Overall, the fig-
ure shows the differences of three paradigms, which indi-
rectly proves the validity of our division of CI.
Figure 3 shows the changes of the proportion of papers
under the three paradigms over time. Each bar in the fig-
ure represents the proportion in each time period, and dif-
ferent colors represent different paradigms. Overall, with
the passage of time, research of isolation paradigm has
decreased, and research of collaboration and feedback
paradigms has gradually increased. This shows that CI
research is increasingly inclined to complex interactions and
feedback.
170216
F. He et al.: Collective Intelligence: A Taxonomy and Survey
III. ISOLATION PARADIGM
In CI of the isolation paradigm, there are two aggregation
methods for output results of independent individuals: simple
statistics such as mean, median and mode, and complex text
mining technology. A summary of the representative studies
on the isolation paradigm is given in Table 2.
A. SIMPLE STATISTICS
A large number of experiments indicate that using certain
simple statistics such as median, mean or mode at the col-
lective level can yield better estimations or decision-making
results than those of most individuals.
Median. Statistician Galton [13] gave the earliest case of
this kind of CI in 1907. Galton analyzed the estimates of
the weight of a particular living ox by 787 different persons
and found that the median of estimates was closer to the true
value than most individual estimates. The median is usually
used for performance analysis with other statistics, especially
mean values in the following studies. For example, for the
task of estimating the number of marbles in a large glass jar,
Krause et al. [14] found that the mean and median values
of collective estimates were 553.6 and 516 respectively, and
came within 1.5% and 8.2% of the true value of 562. Addi-
tionally, Agnew [15] compared the performances of mean,
median and Bayesian method in forecasting procedures for
VOLUME 7, 2019
F. He et al.: Collective Intelligence: A Taxonomy and Survey
macroeconomic indicators such as Gross National Product
(GNP) growth rate and Real Gross National Product (RGNP)
growth rate. Moreover, McNees [16] compared the forecast-
ing performance of mean and median in seven macroeco-
nomic indicators like GNP, unemployment rate and corporate
profits. From the previous experiments, the median has com-
parable performance in forecasting.
The reason that the median takes effect in CI is that no mat-
ter what the real value is, the median of collective forecasts
is closer to the real value than at least 50% of the individuals.
The 50% maximum exists only when the actual outcome xT
is outside of this range [16]:
    
xM − xL xH − xM
xM − , xM + (1)
2 2
In Formula 1, xM , xH and xL are the median, maximum and
minimum of collective estimates, respectively.
Mean. The mean of collective estimates can also improve
the accuracy compared with that of individuals. Zarnowitz
[17] forecasted GNP, RGNP, Implicit Price Deflator (IPD)
and Consumer Expenditures for Durable Goods (CEDG) with
79 individuals and determined that the group mean forecasts
are more accurate than most of the corresponding sets of
individual forecasts. Stock and Watson [18] forecasted the
output growth of seven countries from 1959 to 1999, such
as the United States, Canada and France, by comparing a
series of statistics like mean, median and Trimmed Mean
[38]. They found that simple mean values often yield better
results than complex statistics, which means they have lower
Mean Squared Forecast Errors. Genre et al. [19] explored
multiple combination schemes of expert forecasts, including
combinations based on principal components and trimmed
means, performance-based weighting as well as Bayesian
shrinkage. Only a few of the schemes outperform the mean
of the collective forecasts. Moreover, in forecasting, even the
mean of multiple estimates from an individual also shows a
CI effect [20]–[22].
Mean values are affected by extreme values, which might
cause weak CI robustness. Therefore, some more sophisti-
cated calculation methods for mean values have been intro-
duced, such as Weighted Mean, Trimmed Mean, and Win-
sorized Mean [39]. These statistics can achieve better perfor-
mance in particular forecasting tasks [23], [24].
That the mean of collective estimates is more accurate than
that of the individual estimates can be explained by the Diver-
sity Prediction Theorem [6]. If there are n individual estimates
x1 , x2 , . . . , xi , . . . , xn and a real value xT , then the Collective
Error (CE) is (xT − x̄)2 , which is thePSquared Deviation (SD)
of the estimates’ mean value x̄ = n1 ni=1 xi and the real value
xT . Group P Diversity (GD) is the variance of the estimates,
which is n1 ni=1 (xi − x̄)2 , and Average Individual Error (AIE)
is the average of Squared Deviation
Pbetween the individual
estimates and real value, which is n1 ni=1 (xi −xT )2 . Diversity
Prediction Theorem can be expressed as:
CE = AIE − GD. (2)
VOLUME 7, 2019
According to this theorem, higher diversity leads to lower
collective error and promotes the performance of CI.
Mode. The mode of the individual results in a group
(also called Majority Voting) also achieves better decision
outcomes than a single individual. Ralf et al. [25] analyzed
a dataset involving more than 20,000 diagnoses from more
than 140 doctors and found that the diagnostic performance
of the mode is better than that of the best doctor in the group
when the diagnostic accuracy is relatively similar. Hastie and
Kameda [26] provided an evaluation of nine statistical values
in CI in a simulated test bed environment and determine that
the mode has better robustness.
In certain annotation tasks performed by crowdsourcing,
the majority voting strategy is used to improve the anno-
tation quality. For example, Snow et al. [27] investigated
five annotation tasks of natural language processing (NLP):
affect recognition, word similarity, recognizing textual entail-
ment, temporal event recognition, and word sense disam-
biguation. They found that using the majority voting strategy
only requires a small number of nonexperts to achieve the
annotation quality of experts. Sheng et al. [28] used repeated
labeling and combined the majority voting strategy with
other annotation strategies, such as round-robin and selective
repeated labeling, to improve the quality of annotation. Tian
and Zhu [29] demonstrated a max-margin majority voting for
improving the discriminative ability of majority voting and
further proposed a Bayesian generalization to gain true labels
from the labels of noisy observations.
These phenomena are supported by Condorcet’s Jury The-
orem [7]. This theorem states that if individuals in a group
make decisions independently and the probability of correct-
ness is greater than 0.5, then the probability of making a cor-
rect decision based on the majority voting strategy increases
with the growth of group size.
B. TEXT MINING
By mining the text data of independent individuals, new
knowledge unknown to any individual can be found. This
type of CI usually has two characteristics. First, the individ-
uals are highly distributed in time and space without obvious
interactions. Second, individuals can not perceive the goals at
the collective level.
The most representative study of this type is from the infor-
mation scientist Swanson. Swanson et al. [30] proposed a
new strategy to facilitate the discovery of medical knowledge
based on text mining. A literature set LAB demonstrates the
relationships between topic A and B, and another literature set
LBC demonstrates the relationships between topic B and C.
If LAB and LBC satisfy these two conditions, the relationships
between A and C have certain scientific meaning.
1) Complementary: The new useful information can be
inferred from LAB and LBC together but cannot be obtained
from LAB or LBC alone.
2) Noninteractive: There is no common literature between
LAB and LBC and no citation or co-citation relationship
between them either.
170217

TABLE 3. Some representative studies on collaboration paradigm.
Using the previous strategy, Swanson [31] found a rela-
tionship between Raynaud disease and eicosapentaenoic acid,
which is the main component of fish oil. He also discovered
some medical information such as the relationship between
migraines and magnesium [32]. Swanson and his colleague
Neil Smalheiser [33] then developed a software named
Arrowsmith [40] to identify relationship between Medline
literatures. Other scientists promoted similar research. For
instance, Hristovski et al. [34], [35] developed a literature-
based discovery system BITOLA [41] using an association
mining algorithm and figured out the relationship between
insulin and Huntington disease. Weeber [36], [37] used this
knowledge discovery model to develop a Disease-Adverse
drug reaction-Drug (DAD) system for drug discovery pro-
cess. According to the therapeutic characteristic B, the DAD
can build the pathway from drug A and disease C.
In this kind of CI, every literature in LAB and LBC is a result
of individual behaviors. The aggregation method f refers to
the text mining algorithm or tool. The pathway A ←→ C
corresponds to the CI. This pathway does not exist in any
single literature from LAB and LBC .
There are still two drawbacks in the research of CI based
on the isolation paradigm.
1) In the study using simple statistics as the aggregation
method, single values of the individual output are used for
point estimation, which means the output from the collective
level is also a value. These single values reduce the flexibility
of this kind of aggregation method. In many circumstances,
the output of individuals or groups is uncertain and can be
expressed as an interval or a probability distribution. The
research on CI in these circumstances is still in its infancy.
170218
F. He et al.: Collective Intelligence: A Taxonomy and Survey
2) There are two limitations in research using text mining
in the aggregation method. First, generating a literature set,
such as LAB , mainly depends on the co-occurrence of topics in
documents. Specifically, if topic A and B appear in every doc-
ument in LAB , then we consider these topics to be connected.
However, this assumption is not reliable, so it will affect the
credibility of the final result. Second, the generation of liter-
ature sets and association mining requires high overhead and
tremendous manual work [42], leading to low extensibility.
IV. COLLABORATION PARADIGM
In the CI of the collaboration paradigm, individuals accom-
plish assigned tasks by collaboration, competition and infor-
mation sharing. Individuals can be in the same space-time
domain or in Internet conditions across space-time. For the
CI of the collaboration paradigm, the research focuses on
the analysis and recognition of the key factors, including
social influence, diversity and other factors. A summary of
the representative studies on the collaboration paradigm is
given in Table 3.
A. SOCIAL INFLUENCE
Social influence refers to the transformation of opinions,
attitudes and judgments of an individual from the influence
of other individuals in a group. This is a significant factor of
the collaboration paradigm.
Most studies have proposed that social influence pro-
motes CI. Gürçay et al. [43] analyzed the impacts of social
influence on the estimation accuracy, consensus and con-
fidence of uncertain quantity estimates, and found that the
utilizing of social influence on estimates achieves higher
VOLUME 7, 2019
F. He et al.: Collective Intelligence: A Taxonomy and Survey
accuracy and greater consensus than independent estimates.
Bernstein et al. [44] compared the effect of intermittent and
constant social influence on CI. They found that intermittent
social influence cannot only develop effective information
exchanging and mutual learning, but also maintain a high
level of exploration improving CI. In a lie detection task,
Klein and Epley [45] demonstrated that the results from group
discussion are more accurate than those from individuals
in the isolation paradigm based on majority voting (mode
value). This demonstration verifies that social influence can
increase CI in certain conditions.
However, some studies suggest that social influence
might reduce individual diversity and negatively impact CI.
Lorenz et al. [6] confirmed the assumption that the knowl-
edge of estimates undermines CI by three effects: social
influence, range reduction and confidence. The social influ-
ence effect diminishes the diversity of individual opinions.
Muchnik et al. [46] observed the social bias effect caused by
social influence, especially positive social influence, which
significantly increases the likelihood of positive ratings. For
example, if an up-vote is first inserted into an online forum,
the influence bias of group decision-making will increase
to 25% on average. Becker et al. [47] determined that in
decentralized networks, social interactions between individ-
uals enhance the performance of CI, whereas in centralized
conditions, the influence of central individuals on other indi-
viduals reduces the CI effect.
B. DIVERSITY
Diversity, which is an important factor in CI, refers to the
degree of variation between the individuals in a group. Many
studies have indicated that diversity can improve CI of the
collaboration paradigm. Hong and Page [48] compared the
performance for two kinds of problem-solving teams. One is
a team of randomly selected agents, composed of individuals
with functional diversity, and another is a team comprised
of the best-performing agents. They found that the former
team outperformed the latter team, which is called diversity
trumps ability. Jeffery [49] examined the effect of interper-
sonal congruence as a moderator in the relationship between
diversity and group effectiveness. Interpersonal congruence
refers to the coherence degree to which group members see
the others in the group as the others see themselves. The
investigation shows that diversity tends to improve the per-
formance in groups with high interpersonal congruence but
undermines the performance of groups with low interpersonal
congruence. Aggarwal and Woolley [50] found that CI has
an inverted U-shaped relationship with cognitive divergence.
This discovery shows that regarding the groups of similar
cognitive styles or quite different cognitive styles, the perfor-
mance of CI is maximized when the cognitive divergence is
moderate.
Excessive diversity leads to gaps in communication
and collaboration between individuals and obstructs CI.
Knight et al. [51] investigated that diversity in terms of func-
tional background, age, education and employment tenure of
VOLUME 7, 2019
individuals from Top Management Teams (TMT) negatively
affects on strategic collective consensus. Aggarwal et al. [52]
investigated how the team members’ cognitive styles related
to object and spatial visualization affect the team’s strate-
gic focus and strategic consensus. The results indicated that
diversity in cognitive style is negatively associated with the
formation of group consensus, further affecting the CI.
There are two points need explaining: 1) The functions
of diversity are different in the collaboration paradigm and
isolation paradigm. For the collaboration paradigm, the effect
of diversity represents the complementary views, skills and
knowledge among individuals, which is more conducive to
solving complex tasks. For the latter paradigm, diversity can
reduce the collective error. 2) The effects of social influ-
ence and diversity on CI are not isolated. According to
Mavrodiev et al. [53], social influence does not directly affect
CI, but benefits or harms the CI indirectly by the diversity and
collective error.
C. OTHER FACTORS
Except for social influence and diversity, some researchers
worked on other factors of CI in the collaboration
paradigm, including individual attributes, group composi-
tion, leadership and incentive mechanisms. For example,
Woolley et al. [54] showed that CI is weakly correlated with
the average or maximum individual intelligence of group
members but is strongly correlated with the average social
sensitivity of individuals, the equality in the distribution of
conversational turn-taking, and the proportion of females
in the group. In the Polymath1 project [55], Tim Gowers
and other online netizens tried to find combinatorial proof
of the Density Hales-Jewett Theorem with a side length
of three. This task was accomplished in only 6 weeks and
within 1000 comments. Cranshaw and Kittur [56] analyzed
this collaborative process and found that: 1) Only a small
percentage of the individuals contribute significantly to CI.
2) Leadership plays a critical role in CI. Mann and Helbing
[57] proposed an incentive scheme by rewarding accurate
minority predictions to achieve diversity in individual esti-
mates and increase the accuracy of collective estimates.
Geifman and Raban [58] studied the role of individual self-
efficacy in solving cooperative problems. Self-efficacy is the
subjective judgment of an individual on whether he or she can
use his or her own skills to accomplish a certain task.
At present, there are still two shortcomings in the research
of CI in the collaboration paradigm.
First, the studies available use correlation analysis, stan-
dardized regression coefficients, linear regression, Lasso
regression and other strategies to identify variables that have
correlations with or dependence on CI. However, these meth-
ods cannot demonstrate the causal relationship between vari-
ables and CI, and struggle to explain the causes of CI.
Second, the research on the negative effects of social influ-
ence and the corresponding solutions is still in infancy. The
asynchrony of social effects between individuals greatly dis-
torts collective decisions by introducing social biasing effects
170219

TABLE 4. Some representative studies on feedback paradigm.
including herding and snowballing [59]. The herding effect
indicates that an individual in a group follows the opinions
of the population majority. The snowballing effect means if
an opinion earns an initial advantage, the influence becomes
increasingly obvious.
V. FEEDBACK PARADIGM
The CI in the feedback paradigm is actually an emergence
phenomenon. On the one hand, simple individual collab-
orative behavior at the micro-level can influence variables
at the macro-level and CI-related variables. On the other
hand, macro-level variables restrict or promote the micro-
level through feedback mechanisms. With this iteration pro-
cess, a new characteristic or structure of CI at the macro-level
can be established.
The following subsections §V.A and §V.B present research
analysis on CI of the feedback paradigm in social animals and
human societies respectively. These studies are summarized
in Table 4.
A. COLLECTIVE INTELLIGENCE EMERGENCE IN SOCIAL
ANIMALS
Since the 1990s, a series of CI algorithms for resolving
optimization problems has been proposed by considering
social animals such as ants, bees and birds, which concentrate
on the cooperative behavior of groups and the emergence
mechanism of CI.
The earliest algorithm of this type is Ant Colony Opti-
mization (ACO) Algorithm in 1992 proposed by Dorigo [60].
ACO algorithm simulated the process of how ants find the
most effective path between the nest and a food source.
In this algorithm, the environment Et refers to the amount
of pheromone at time t between two positions (or status),
which is generated by the behaviors of ant colonies before
time t. The individual ant ai generates indirect interactions
based on the environment Et , that is, the transition probability
is calculated according to Et , and the next position (or status)
is transferred to other individual ants until all the positions
are traversed. The behavior output of individual ai at time t
refers to the amount of pheromone xi,t between two positions
170220
F. He et al.: Collective Intelligence: A Taxonomy and Survey
(or status) during the traversal process. At the collective level,
f (Et , Xt ) calculates the amount of pheromone Et+1 at time
t + 1, in which the feedback mechanism is represented by
influencing Xt+1 at time t + 1. The CI eventually obtains the
path described by the pheromones.
Kennedy and Eberhart [61] then proposed Particle Swarm
Optimization (PSO) algorithm in 1995, which simulated birds
flocking. The individuals in a flock (also called particles)
have two attributes: velocity and position. At the individual
level, the velocity vi,t of individual ai at time t is updated
based on three variables (usually in vector form), vi,t−1 ,
the best position of the individual (pbesti ) and the global
optimal position of the group (gbest). gbest can be regarded
as the environment E for individual interactions. The position
xi,t at time t, which is the individual output, is updated on
the basis of xi,t−1 and vi,t . At the group level, pbesti (i ∈
[1, . . . , n]) and gbest are updated according to the value of Xt ,
and affect Xt+1 at time t + 1. After the algorithm converges,
the output gbest is treated as CI.
An algorithm named Artificial Bee Colony (ABC) was
proposed by Karaboga and Basturk [62], which is used to
solve the optimization problem of a multi-variable function.
In ABC algorithm, individuals can be divided into three roles:
employed bees, onlooker bees and scout bees. They can trans-
fer their roles in particular situation. ABC algorithm finds the
global optimal solution through individual local optimization.
At the individual level, the employed bees search for new food
sources in the neighborhood of the current source and eval-
uate the fitness. Onlooker bees select a honey source based
on information shared by the employed bees and calculate
the fitness value as well. If the fitness of the food source
has not increased after multiple iterations, the corresponding
employed bees change to onlooker bees and randomly gen-
erate a new position to replace the current food source. The
output of individual ai at time t is a tuple xt = (si,t , fit(si,t )),
si,t refers to the position of the current food source of ai at
time t and fit(si,t ) is the fitness of si,t . Contrasting Xt and
Xt+1 , it shows that a rich source attracts more individuals by
positive feedback at the collective level, while a poor source
is gradually abandoned.
VOLUME 7, 2019
F. He et al.: Collective Intelligence: A Taxonomy and Survey
In recent years, people have proposed a series of more
complicated CI algorithms based on the behavior of social
animals. Mirjalili et al. [63] raised the Grey Wolf Optimizer
(GWO) algorithm to represent the hierarchy of members. The
GWO algorithm simulates the leadership hierarchy of four
types of grey wolves, including alpha, beta, delta and omega,
and also simulates three steps of prey hunting, involving
searching, encircling, and attacking. The GWO algorithm
achieves competitive results on 29 well-known test functions.
Meng et al. [64] proposed an algorithm named Bird Swarm
Algorithm based on social behaviors in bird swarms, such as
foraging, vigilance and flight behavior. The effectiveness of
bird swarm algorithms has been verified by comparison tests
on 18 benchmark problems. In addition to social animals,
people found that the swarm behaviors of lower organisms,
such as bacteria and physarum, also exhibit CI. The cor-
responding algorithms have been designed [65], [66]. This
kind of algorithm is supposed to be robust, decentralized and
parallelized.
A synergetic approach can be used to describe and explain
this kind of CI. Synergetics is an interdisciplinary research
area that illustrates how subsystems in open complex sys-
tems form structures of space, time, or function through self-
organization [10].
Utilizing a synergetic approach to interpret this kind of
CI mechanism can be treated as a process of finding order
parameters. Generally, there are a large number of variables
in a mathematical model used to describe a complex system,
but most are effective only for a short time and have little
influence on the evolution of the system. These variables
are called fast variables. Another kind of variables are called
order parameters, also known as slow variables. This kind of
variables indicate the degree of macro-order in a system. They
are the total contribution of the synergetic behavior from each
subsystem or individual and control the transformation of fast
variables. The order parameters play considerable roles in
the system evolution. Usually, there are fewer order param-
eters in the system, and sometimes only one. For example,
in the previous ACO, PSO and ABC algorithms, the order
parameters are the amount of pheromone, the position of
particles and the fitness of honey source, respectively. The
adiabatic elimination procedure [69] can usually be used to
eliminate an abundance of fast variables to distinguish the
order parameters.
B. COLLECTIVE INTELLIGENCE EMERGENCE IN HUMAN
SOCIETY
There also exists CI of feedback paradigm in human society,
which is manifested as self-interested and dispersed inter-
actions of individuals at the micro-level, and orderly social
phenomena emerging at the macro-level, such as Invisible
Hand [70] in market economy and Spontaneous Order [71]
in human society.
Yukalov et al. [67] proposed an evolution equation to
describe a society composed of different types of individ-
uals, including cooperators, defectors, and regulators. The
VOLUME 7, 2019
evolution equation can explain how the interactions between
individuals of different types at the micro-level can develop
a structurally stable society at the macro-level. The authors
ranked ten countries with this model and found a remarkable
concordance with certain standard economic rankings.
Tao [68] used Boltzmann distribution with Marginal
Labor-Capital Return and Marginal Technology Return as
parameters to describe the enterprise income under the com-
petitive economy. Based on the differential equation of GDP,
he deduced that the called technological level is a self-
referential variable that can be considered to be an index
measuring the CI in an economic environment. The com-
petitive behavior of enterprises helps a higher technological
level to emerge, and the technological level controls or guides
enterprise behaviors by a feedback mechanism.
The present studies aiming at the emergence mechanisms
of CI under the feedback paradigm mainly focus on groups
with simple rules and slight discrepancies between individ-
uals. Many groups in reality, especially human society, have
complex individual behaviors and great diversity. Although
theoretically we could demonstrate and model them by a syn-
ergetic method, it would still be hard to promote quantitative
analysis because of the introduction of a large number of
variables and differential equations.
VI. RECENT APPLICATIONS OF CI
Many CI-related applications have been proposed to solve
practical problems. We discuss certain typical works
in 2019 in this section, where each work can be categorized
appropriately to one of the three paradigms, with the hope
of helping researchers understand the current application
scenarios of CI. These applications with contents and the
corresponding paradigms are shown in Table 5.
A. APPLICATIONS OF THE ISOLATION PARADIGM
For the isolation paradigm, there are several applica-
tions focusing on providing intelligent services by col-
lective decisions of independent individuals. For example,
Dambanemuya and Horvát [72] aggregated personal lending
behaviors and successful loan payments, which can be seen
as individual behavior results, to provide reliable decision
support. This can overcome P2P’s weakness of information
asymmetry about borrowers’ credibility. Lee et al. [73] pro-
posed a pest diagnosis system using data acquisition and
deep learning through CI. The required data, in the prediction
of pests arising from the plant cultivation, is independently
collected from experts and user participates. By aggregating
the data from a large number of surveyors, a rapid diagnosis
can be made.
B. APPLICATIONS OF THE COLLABORATION PARADIGM
There are many CI-related applications of the collabora-
tion paradigm and we choose six of them as examples.
Otoum et al. [74] presented a restricted Boltzmann
machine-based clustered intrusion detection system (RBC-
IDS) in the monitoring of critical infrastructures through
170221

TABLE 5. Some applications of CI in 2019.
wireless sensor networks (WSNs). In RBC-IDS, a selection
method is accomplished to integrate information, which cal-
culates the weight of each sensor node and compares it with
the weights of other nodes. In densely crowded environments,
Aloqaily et al. [75] described how to replicate data from the
cloud to the edge, and then to mobile devices to provide
faster data access for users. Through node collaboration, data
replication and service composition offload some workload
burdens from the cloud. Kotb et al. [76] proposed a workflow-
net based framework for efficient cloud computing, through
which fog computing devices are able to form a cooperative
Internet of Things (IoT) service delivery system and enhance
performance gains of cloud computing. Yeh et al. [77]
focused on the case of modeling trajectory data collected
from multi-player sports games, namely basketball and soc-
cer, where each player is treated as an agent. They leveraged
graph neural networks (GNNs) and variational recurrent
neural networks (VRNNs) to process interaction information
between the agents and then perform conditional prediction.
Belbachir et al. [78] proposed a self-adaptive mechanism
for traffic regulation based on cooperative agents, which can
minimize traffic congestion. The cooperative behaviors are
not pre-defined, but they emerge from the agent interactions
at a local level. Awal et al. [79] predicted user preferences
and future behaviors by embedding collective knowledge
in social networks. Concretely, they presented an extension
of the signed modularity and determined users’ degrees of
affiliation towards various communities. The signs (posi-
tive or negative) of the links are considered for collective
behavior prediction in signed social networks.
C. APPLICATIONS OF THE FEEDBACK PARADIGM
There are several CI-related recent applications of the feed-
back paradigm, which mainly integrate certain advanced
algorithms, such as reinforcement learning and ACO,
170222
F. He et al.: Collective Intelligence: A Taxonomy and Survey
to simulate the interactions between individuals and the envi-
ronment. Otoum et al. [80] proposed a reinforcement learn-
ing method for intrusion detection system (RL-IDS). With
RL-IDS, the agents (sensors in WSNs) interpret the present
state of the environment and fulfill an action that eventually
results in certain rewards to the agents. The experimental
results show that RL-IDS achieves state-of-the-art perfor-
mance. Petrillo et al. [81] proposed Swarm Debugging Infras-
tructure (SDI), which can collect data about developers’ inter-
active debugging sessions, and also feedback recommenda-
tions to developers. Such a continuous iterative cycle reaches
the target of swarm debugging. Boveiri et al. [82] proposed
a high-performance approach based on the MaxâĂ"Min Ant
System (MMAS), which is an efficient variation in the family
of ACO algorithms. In MMAS, optimal task-order can be
achieved as a feedback by scheduling in homogeneous multi-
processor environments. Therefore, MMAS can utilize local
mini data-centers to release the burden from the main data-
center.
In summary, recent CI-related applications, especially
those of the collaboration and feedback paradigms, have
been widely applied in various scenarios, which meet the
results of literature analysis in Section II. Some mainstream
models or methods of artificial intelligence have been used in
CI, such as deep learning and reinforcement learning.
VII. FUTURE RESEARCH DIRECTIONS
In this section, we summarize six future research directions
related to three paradigms of CI.
Nonnumerical Estimation under Isolation Paradigm.
At present, in CI of the isolation paradigm, statistics are
mainly used for numerical estimation. The results of indi-
vidual estimation are usually nonnumerical, such as inter-
val and probability distributions. Simply treating the results
as a numerical value eliminates uncertainty but causes
VOLUME 7, 2019
F. He et al.: Collective Intelligence: A Taxonomy and Survey
information loss [28]. In addition, at the group level, peo-
ple also want to obtain nonnumerical aggregation results.
Therefore, the aggregation model oriented to nonnumerical
estimation should be focused on.
Causality Analysis in Collective Intelligence. Correla-
tion analysis is mainly used to analyze the influencing factors
of collective intelligence, but there is still a lack of studies
on causality analysis. Causality refers to the interactions
between variables, and the change of one variable is triggered
by another. Although correlation analysis is significant in the
mining of causality, it is neither necessary nor sufficient to
establish causation [83]. Moreover, causality is an important
foundation for restraining the negative effects and improving
the CI. There are still some problems to be solved: (1) Causal-
ity map and recognition of latent variables in CI scenarios; (2)
Control and regulation of CI based on causal inference.
Human-Computer Hybrid Collective Intelligence.
Actually, many engineering problems might involve intelli-
gent computing tasks under uncertain and open conditions.
The human brain is better at dealing with such tasks than
the available artificial intelligence. For this reason, human
computing [84] and human-in-the-loop augmented intelli-
gence [85] have been proposed, aiming at studying how to use
humans as computing components and solving complex prob-
lems in reality by combining humans and computers. From
the perspective of CI, this kind of work could also be consid-
ered to provide solutions for complex tasks through human-
computer collaboration in heterogeneous groups. The tasks
are difficult to accomplish by only human beings or com-
puters. The main problems include: (1) Model and emer-
gence mechanism of human-computer hybrid CI; (2) Human-
computer perception and understanding under uncertain and
open conditions; (3) Task modeling and dynamic allocation;
(4) Measurement of human-computer hybrid CI.
Synergetics Model of Networked Collective Intelli-
gence. The Internet breaks the barrier of physical space-time
on human group collaboration and helps to form more com-
plex networked CI (NCI) in larger scales of time and space.
This group has the attributes of large individual differences
(interest, ability, and diversity of objectives), high dynamic
(the joining and exiting of members are not constrained), and
cross-temporal distribution, which greatly improves the com-
plexity of modeling. The main problems to be solved include:
(1) The evolutionary model of NCI; (2) The identification
of order parameters and the construction of order parameter
equations; (3) The application in Computer-Supported Coop-
erative Work (CSCW) [86], [87] and Computer-Supported
Cooperative Learning (CSCL) [88], [89].
Discrimination and Elimination of Dysergy Phe-
nomenon. CI is still unable to emerge in most cases in human
society even with the characteristics of interactions and feed-
back. From the perspective of synergetics, the main reason
is the existence of the dysergy phenomenon in groups [90].
Dysergy refers to the interactions between individuals that
promote one function but impede another. In order to ensure
the emergence and sustainability of CI, it is necessary to work
VOLUME 7, 2019
on: (1) How to discriminate dysergy phenomena in the pro-
cess of individual interactions; (2) How to eliminate or relieve
dysergy phenomena on CI.
Measurement of Collective Intelligence. There are stud-
ies showing that groups also have the similar factor of general
intelligence, called g factor, of individuals dealing with mul-
tiple cognitive tasks. Thus far, there have been some studies
on the measurement of CI. Woolley et al. [54] found that the
level of CI is weakly correlated with the average or maximum
intelligence of individuals in a group. Kosinski et al. [91]
proposed the intelligence level of a group, called Crowd
IQ, by a method similar to an IQ questionnaire. However,
there are still some problems to be solved: (1) Measurement
indicators of CI in different scenarios, such as prediction
and collaborative learning; (2) Real-time measurement and
dynamic regulation of CI.
VIII. CONCLUSION
CI is a long-standing and vast research field, involving soci-
ology, economics, biology, artificial intelligence and other
different fields. The overlapping or closely related directions
include swarm intelligence, wisdom of the crowd, human
computation, and crowdsourcing. In this survey, we define CI
as the intelligence that emerges from the macro-level of a col-
lection and transcends that of the individuals. Additionally,
we try to provide an overview of the research that matches
with this definition. First, we demonstrate a taxonomy of CI
under three paradigms: isolation, collaboration and feedback,
depending on the existence of interactions between individu-
als and the feedback mechanism in the aggregation process.
Second, from the perspective of statistical literature analysis,
we explain the differences among these three paradigms and
their development in recent years. Third, we analyze the CI
studies of each paradigm and explain the generation mech-
anisms and theoretical bases of the different types. Fourth,
we describe certain CI-related applications in 2019, utiliz-
ing different techniques to address some frontier challenges
in different areas. Finally, we summarize the prospective
research directions of CI.
We hope that this survey provides a guideline for beginners
in CI, helps them classify existing studies and sheds light on
future research.
REFERENCES
[1] P. Michelucci and J. L. Dickinson, ‘‘The power of crowds,’’ Science,
vol. 351, no. 6268, pp. 32–33, 2016.
[2] T. Yan, M. Marzilli, R. Holmes, D. Ganesan, and M. Corner, ‘‘mCrowd:
A platform for mobile crowdsourcing,’’ in Proc. 7th ACM Conf. Embedded
Netw. Sensor Syst., 2009, pp. 347–348.
[3] C. Yu, Y. Chai, and Y. Liu, ‘‘Literature review on collective intelligence:
A crowd science perspective,’’ Int. J. Crowd Sci., vol. 2, no. 1, pp. 64–73,
2018.
[4] J. Salminen, ‘‘Collective intelligence in humans: A literature review,’’
2012, arXiv:1204.3401. [Online]. Available: https://arxiv.org/abs/
1204.3401
[5] J. Krause, G. D. Ruxton, and S. Krause, ‘‘Swarm intelligence in animals
and humans,’’ Trends Ecol. Evol., vol. 25, no. 1, pp. 28–34, 2010.
[6] J. Lorenz, H. Rauhut, F. Schweitzer, and D. Helbing, ‘‘How social influ-
ence can undermine the wisdom of crowd effect,’’ Proc. Nat. Acad. Sci.
USA, vol. 108, no. 22, pp. 9020–9025, 2011.
170223

[7] D. Berend and J. Paroush, ‘‘When is condorcet’s jury theorem valid?’’
Social Choice Welfare, vol. 15, no. 4, pp. 481–488, 1998.
[8] M. Nielsen, Reinventing Discovery: The New Era of Networked Science.
Princeton, NJ, USA: Princeton Univ. Press, 2011.
[9] (2017). Synergetics (Haken). Accessed: Oct. 16, 2019. [Online]. Available:
https://en.wikipedia.org/w/index.php?title=Synergetics_(Haken)&oldid=
805750827
[10] H. Haken, ‘‘Synergetics—An outline,’’ in Synergetic Computers and Cog-
nition. Berlin, Germany: Springer, 2004, pp. 20–35.
[11] W. Cui, Y. Wu, S. Liu, F. Wei, M. X. Zhou, and H. Qu, ‘‘Context preserving
dynamic word cloud visualization,’’ in Proc. IEEE Pacific Vis. Symp.
(PacificVis), Mar. 2010, pp. 121–128.
[12] F. Heimerl, S. Lohmann, S. Lange, and T. Ertl, ‘‘Word cloud explorer: Text
analytics based on word clouds,’’ in Proc. 47th Hawaii Int. Conf. Syst. Sci.,
2014, pp. 1833–1842.
[13] F. Galton, ‘‘Vox populi,’’ Nature, vol. 75, no. 7, pp. 450–451, 1907.
[14] S. Krause, R. James, J. J. Faria, G. D. Ruxton, and J. Krause, ‘‘Swarm intel-
ligence in humans: Diversity can trump ability,’’ Animal Behav., vol. 81,
no. 5, pp. 941–948, 2011.
[15] C. E. Agnew, ‘‘Bayesian consensus forecasts of macroeconomic vari-
ables,’’ J. Forecasting, vol. 4, no. 4, pp. 363–376, 1985.
[16] S. K. McNees, ‘‘The uses and abuses of ‘consensus’ forecasts,’’ J. Fore-
casting, vol. 11, no. 8, pp. 703–710, 1992.
[17] V. Zarnowitz, ‘‘The accuracy of individual and group forecasts from busi-
ness outlook surveys,’’ J. Forecasting, vol. 3, no. 1, pp. 11–26, 1984.
[18] J. H. Stock and M. W. Watson, ‘‘Combination forecasts of output growth
in a seven-country data set,’’ J. Forecasting, vol. 23, no. 6, pp. 405–430,
Sep. 2004.
[19] V. Genre, G. Kenny, A. Meyler, and A. Timmermann, ‘‘Combining expert
forecasts: Can anything beat the simple average?’’ Int. J. Forecasting,
vol. 29, no. 1, pp. 108–121, 2013.
[20] E. Vul and H. Pashler, ‘‘Measuring the crowd within probabilistic repre-
sentations within individuals,’’ Psychol. Sci., vol. 19, no. 7, pp. 645–647,
2008.
[21] D. van Dolder and M. J. van den Assem, ‘‘The wisdom of the inner crowd
in three large natural experiments,’’ Nature Hum. Behav., vol. 2, no. 1,
pp. 21–26, 2018.
[22] S. M. Herzog and R. Hertwig, ‘‘The wisdom of many in one mind:
Improving individual judgments with dialectical bootstrapping,’’ Psychol.
Sci., vol. 20, no. 2, pp. 231–237, 2009.
[23] V. R. R. Jose and R. L. Winkler, ‘‘Simple robust averages of forecasts:
Some empirical results,’’ Int. J. Forecasting, vol. 24, no. 1, pp. 163–169,
2008.
[24] D. V. Budescu and E. Chen, ‘‘Identifying expertise to extract the wisdom
of crowds,’’ Manage. Sci., vol. 61, no. 2, pp. 267–280, 2015.
[25] R. H. J. M. Kurvers, S. M. Herzog, R. Hertwig, J. Krause, P. A. Carney,
A. Bogart, G. Argenziano, I. Zalaudek, and M. Wolf, ‘‘Boosting medical
diagnostics by pooling independent judgments,’’ Proc. Nat. Acad. Sci.
USA, vol. 113, no. 31, pp. 8777–8782, 2016.
[26] R. Hastie and T. Kameda, ‘‘The robust beauty of majority rules in group
decisions,’’ Psychol. Rev., vol. 112, no. 2, p. 494, 2005.
[27] R. Snow, B. O’Connor, D. Jurafsky, and A. Y. Ng, ‘‘Cheap and fast—
But is it good?: Evaluating non-expert annotations for natural language
tasks,’’ in Proc. Conf. Empirical Methods Natural Lang. Process., 2008,
pp. 254–263.
[28] V. S. Sheng, F. Provost, and P. G. Ipeirotis, ‘‘Get another label? Improving
data quality and data mining using multiple, noisy labelers,’’ in Proc.
14th ACM SIGKDD Int. Conf. Knowl. Discovery Data Mining, 2008,
pp. 614–622.
[29] T. Tian and J. Zhu, ‘‘Max-margin majority voting for learning from
crowds,’’ in Proc. Adv. Neural Inf. Process. Syst., 2015, pp. 1621–1629.
[30] D. R. Swanson and N. R. Smalheiser, ‘‘An interactive system for find-
ing complementary literatures: A stimulus to scientific discovery,’’ Artif.
Intell., vol. 91, no. 2, pp. 183–203, 1997.
[31] D. R. Swanson, ‘‘Fish oil, raynaud’s syndrome, and undiscovered public
knowledge,’’ Perspect. Biol. Med., vol. 30, no. 1, pp. 7–18, 1986.
[32] D. R. Swanson, ‘‘Migraine and magnesium: Eleven neglected connec-
tions,’’ Perspect. Biol. Med., vol. 31, no. 4, pp. 526–557, 1988.
[33] N. R. Smalheiser, ‘‘The Arrowsmith project: 2005 status report,’’ in Proc.
Int. Conf. Discovery Sci. Berlin, Germany: Springer, 2005, pp. 26–43.
[34] D. Hristovski, C. Friedman, T. C. Rindflesch, and B. Peterlin, ‘‘Exploiting
semantic relations for literature-based discovery,’’ in Proc. AMIA Annu.
Symp., 2006, p. 349.
170224
F. He et al.: Collective Intelligence: A Taxonomy and Survey
[35] D. Hristovski, B. Peterlin, J. A. Mitchell, and S. M. Humphrey, ‘‘Using
literature-based discovery to identify disease candidate genes,’’ Int. J. Med.
Informat., vol. 74, nos. 2–4, pp. 289–298, 2005.
[36] M. Weeber, ‘‘Drug discovery as an example of literature-based discovery,’’
in Computational Discovery of Scientific Knowledge. Berlin, Germany:
Springer, 2007, pp. 290–306.
[37] M. Weeber, H. Klein, A. R. Aronson, J. G. Mork, L. de Jong-van Den Berg,
and R. Vos, ‘‘Text-based discovery in biomedicine: The architecture of the
dad-system,’’ in Proc. AMIA Symp., 2000, p. 903.
[38] (2019). Truncated Mean. Accessed: Oct. 16, 2019. [Online]. Available:
https://en.wikipedia.org/w/index.php?title=Truncated_mean&oldid=8882
10015
[39] (2019). Winsorized Mean. Accessed: Oct. 16, 2019. [Online]. Available:
https://en.wikipedia.org/w/index.php?title=Winsorized_mean&oldid=88
1012445
[40] (2005). The Arrowsmith Project Homepage. Accessed: Oct. 16, 2019.
[Online]. Available: http://arrowsmith.psych.uic.edu
[41] (2006). Bitola—Biomedical Discovery Support System. Accessed:
Oct. 16, 2019. [Online]. Available: http://ibmi.mf.uni-lj.si/sl/bitola
[42] W. Pratt and M. Yetisgen-Yildiz, ‘‘LitLinker: Capturing connections across
the biomedical literature,’’ in Proc. 2nd Int. Conf. Knowl. Capture, 2003,
pp. 105–112.
[43] B. Gürçay, B. A. Mellers, and J. Baron, ‘‘The power of social influence on
estimation accuracy,’’ J. Behav. Decis. Making, vol. 28, no. 3, pp. 250–261,
2014.
[44] E. Bernstein, J. Shore, and D. Lazer, ‘‘How intermittent breaks in interac-
tion improve collective intelligence,’’ Proc. Nat. Acad. Sci. USA, vol. 115,
no. 35, pp. 8734–8739, 2018.
[45] N. Klein and N. Epley, ‘‘Group discussion improves lie detection,’’ Proc.
Nat. Acad. Sci. USA, vol. 112, no. 24, pp. 7460–7465, 2015.
[46] L. Muchnik, S. Aral, and S. J. Taylor, ‘‘Social influence bias: A randomized
experiment,’’ Science, vol. 341, no. 6146, pp. 647–651, 2013.
[47] J. Becker, D. Brackbill, and D. Centola, ‘‘Network dynamics of social
influence in the wisdom of crowds,’’ Proc. Nat. Acad. Sci. USA, vol. 114,
no. 26, pp. E5070–E5076, 2017.
[48] L. Hong and S. E. Page, ‘‘Groups of diverse problem solvers can outper-
form groups of high-ability problem solvers,’’ Proc. Nat. Acad. Sci. USA,
vol. 101, no. 46, pp. 16385–16389, 2004.
[49] J. T. Polzer, L. P. Milton, and W. B. Swarm, Jr., ‘‘Capitalizing on diversity:
Interpersonal congruence in small work groups,’’ Administ. Sci. Quart.,
vol. 47, no. 2, pp. 296–324, 2002.
[50] I. Aggarwal, A. W. Woolley, C. F. Chabris, and T. W. Malone, ‘‘Cognitive
diversity, collective intelligence, and learning in teams,’’ in Proc. Collec-
tive Intell. Conf., 2015, vol. 1, nos. 3.1, p. 3.
[51] D. Knight, C. L. Pearce, K. G. Smith, J. D. Olian, H. P. Sims, K. A. Smith,
and P. Flood, ‘‘Top management team diversity, group process, and strate-
gic consensus,’’ Strategic Manage. J., vol. 20, no. 5, pp. 445–465, 1999.
[52] I. Aggarwal and A. W. Woolley, ‘‘Do you see what I see? The effect of
members’ cognitive styles on team processes and errors in task execution,’’
Org. Behav. Hum. Decis. Process., vol. 122, no. 1, pp. 92–99, 2013.
[53] P. Mavrodiev, C. J. Tessone, and F. Schweitzer, ‘‘Effects of social influence
on the wisdom of crowds,’’ 2012, arXiv:1204.3463. [Online]. Available:
https://arxiv.org/abs/1204.3463
[54] A. W. Woolley, C. F. Chabris, A. Pentland, N. Hashmi, and T. W. Malone,
‘‘Evidence for a collective intelligence factor in the performance of human
groups,’’ Science, vol. 330, no. 6004, pp. 686–688, 2010.
[55] T. Gowers and M. Nielsen, ‘‘Massively collaborative mathematics,’’
Nature, vol. 461, no. 7266, p. 879, 2009.
[56] J. Cranshaw and A. Kittur, ‘‘The polymath project: Lessons from a suc-
cessful online collaboration in mathematics,’’ in Proc. SIGCHI Conf. Hum.
Factors Comput. Syst., 2011, pp. 1865–1874.
[57] R. P. Mann and D. Helbing, ‘‘Optimal incentives for collective intelli-
gence,’’ Proc. Nat. Acad. Sci. USA, vol. 114, no. 20, pp. 5077–5082, 2017.
[58] D. Geifman and D. R. Raban, ‘‘Collective problem-solving: The role of
self-efficacy, skill, and prior knowledge,’’ Interdiscipl. J. E-Learn. Learn.
Objects, vol. 11, no. 1, pp. 159–178, 2015.
[59] L. B. Rosenberg, ‘‘Human swarming, a real-time method for parallel
distributed intelligence,’’ in Proc. Swarm/Hum. Blended Intell. Workshop
(SHBI), Sep. 2015, pp. 1–7.
[60] M. Dorigo and T. Stützle, Ant Colony Optimization. Cambridge, MA, USA:
MIT Press, 2004.
[61] J. Kennedy and R. C. Eberhart, ‘‘Particle swarm optimization,’’ in Proc.
Int. Conf. Netw., 1995, pp. 1942–1948.
VOLUME 7, 2019
F. He et al.: Collective Intelligence: A Taxonomy and Survey
[62] D. Karaboga and B. Basturk, ‘‘A powerful and efficient algorithm for
numerical function optimization: Artificial bee colony (ABC) algorithm,’’
J. Global Optim., vol. 39, no. 3, pp. 459–471, Apr. 2007.
[63] S. Mirjalili, S. M. Mirjalili, and A. Lewis, ‘‘Grey wolf optimizer,’’ Adv.
Eng. Softw., vol. 69, pp. 46–61, Mar. 2014.
[64] X.-B. Meng, X. Z. Gao, L. Lu, Y. Liu, and H. Zhang, ‘‘A new bio-inspired
optimisation algorithm: Bird swarm algorithm,’’ J. Exp. Theor. Artif. Intell.,
vol. 28, pp. 673–687, Jun. 2015.
[65] A. Shklarsh, G. Ariel, E. Schneidman, and E. Ben-Jacob, ‘‘Smart swarms
of bacteria-inspired agents with performance adaptable interactions,’’
PLoS Comput. Biol., vol. 7, no. 9, p. e1002177, 2011.
[66] L. Liu, Y. Song, H. Zhang, H. Ma, and A. V. Vasilakos, ‘‘Physarum
optimization: A biology-inspired algorithm for the steiner tree problem in
networks,’’ IEEE Trans. Comput., vol. 64, no. 3, pp. 818–831, Mar. 2015.
[67] V. I. Yukalov, E. P. Yukalova, and D. Sornette, ‘‘Utility rate equations of
group population dynamics in biological and social systems,’’ PLoS ONE,
vol. 8, no. 12, p. e83225, 2013.
[68] Y. Tao, ‘‘Swarm intelligence in humans: A perspective of emergent evolu-
tion,’’ Phys. A, Stat. Mech. Appl., vol. 502, pp. 436–446, Jul. 2018.
[69] (2019). Autler-Townes. Accessed: Oct. 16, 2019. [Online]. Available:
https://en.wikipedia.org/w/index.php?title=Autler%E2%80%93Townes_
effect&oldid=920444858
[70] (2019). Invisible Hand. Accessed: Oct. 16, 2019. [Online]. Available:
https://en.wikipedia.org/w/index.php?title=Invisible_hand&oldid=91469
2334
[71] (2019). Spontaneous Order. Accessed: Oct. 16, 2019. [Online]. Available:
https://en.wikipedia.org/w/index.php?title=Spontaneous_order&oldid=92
1202468
[72] H. K. Dambanemuya and E.-Á. Horvát, ‘‘Harnessing collective intelli-
gence in P2P lending,’’ in Proc. 10th ACM Conf. Web Sci., 2019, pp. 57–64.
[73] H.-Y. Lee, D.-H. Kim, and K.-R. Park, ‘‘Pest diagnosis system based
on deep learning using collective intelligence,’’ Int. J. Elect. Eng. Edu.,
pp. 1–15, 2019.
[74] S. Otoum, B. Kantarci, and H. T. Mouftah, ‘‘On the feasibility of deep
learning in sensor network intrusion detection,’’ IEEE Netw. Lett., vol. 1,
no. 2, pp. 68–71, Jun. 2019.
[75] M. Aloqaily, I. Al Ridhawi, H. B. Salameh, and Y. Jararweh, ‘‘Data
and service management in densely crowded environments: Challenges,
opportunities, and recent developments,’’ IEEE Commun. Mag., vol. 57,
no. 4, pp. 81–87, Apr. 2019.
[76] Y. Kotb, I. Al Ridhawi, M. Aloqaily, T. Baker, Y. Jararweh, and H. Tawfik,
‘‘Cloud-based multi-agent cooperation for IoT devices using workflow-
nets,’’ J. Grid Comput., pp. 1–26, Jun. 2019.
[77] R. A. Yeh, A. G. Schwing, J. Huang, and K. Murphy, ‘‘Diverse generation
for multi-agent sports games,’’ in Proc. IEEE Conf. Comput. Vis. Pattern
Recognit., Jun. 2019, pp. 4610–4619.
[78] A. Belbachir, A. El Fallah-Seghrouchni, A. Casals, and M. Pasin, ‘‘Smart
mobility using multi-agent system,’’ Procedia Comput. Sci., vol. 151,
pp. 447–454, Apr. 2019.
[79] G. K. Awal and K. Bharadwaj, ‘‘Leveraging collective intelligence for
behavioral prediction in signed social networks through evolutionary
approach,’’ Inf. Syst. Frontiers, vol. 21, no. 2, pp. 417–439, 2019.
[80] S. Otoum, B. Kantarci, and H. Mouftah, ‘‘Empowering reinforcement
learning on big sensed data for intrusion detection,’’ in Proc. IEEE Int.
Conf. Commun. (ICC), May 2019, pp. 1–7.
[81] F. Petrillo, Y.-G. Guéhéneuc, M. Pimenta, C. D. S. Freitas, and F. Khomh,
‘‘Swarm debugging: The collective intelligence on interactive debugging,’’
J. Syst. Softw., vol. 153, pp. 152–174, Jul. 2019.
[82] H. R. Boveiri, R. Khayami, M. Elhoseny, and M. Gunasekaran, ‘‘An effi-
cient swarm-intelligence approach for task scheduling in cloud-based
Internet of Things applications,’’ J. Ambient Intell. Humanized Comput.,
vol. 10, no. 9, pp. 3469–3479, 2019.
[83] G. Sugihara, R. May, H. Ye, C.-H. Hsieh, E. Deyle, M. Fogarty,
and S. Munch, ‘‘Detecting causality in complex ecosystems,’’ Science,
vol. 338, no. 6106, pp. 496–500, Oct. 2012.
[84] L. Von Ahn, ‘‘Human computation,’’ in Proc. IEEE 24th Int. Conf. Data
Eng., Apr. 2008, pp. 1–2.
[85] N.-N. Zheng, Z.-Y. Liu, P.-J. Ren, Y.-Q. Ma, S.-T. Chen, S.-Y. Yu,
J.-R. Xue, B.-D. Chen, and F.-Y. Wang, ‘‘Hybrid-augmented intelligence:
Collaboration and cognition,’’ Frontiers Inf. Technol. Electron. Eng.,
vol. 18, no. 2, pp. 153–179, Feb. 2017.
[86] J. Grudin, ‘‘Computer-supported cooperative work: History and focus,’’
Computer, vol. 27, no. 5, pp. 19–26, May 1994.
VOLUME 7, 2019
[87] (2019). Computer-Supported Cooperative Work. Accessed:
Oct. 16, 2019. [Online]. Available: https://en.wikipedia.org/w/index.php?
title=Computer-supported_cooperative_work&oldid=905294828
[88] G. Stahl, ‘‘A view of computer-supported collaborative learning research
today,’’ in Proc. Int. Conf. Collaboration Technol. Syst. (CTS), 2011,
p. 325.
[89] (2019). Computer-Supported Collaborative Learning. Accessed:
Oct. 16, 2019. [Online]. Available: https://en.wikipedia.org/w/index.
php?title=Computer-supported_collaborative_learning&oldid=921034480
[90] N. A. Coulter, Human Synergistics (The Revised Internet Edition). Bergen
County, NJ, USA: The Time-Binders, 2002.
[91] M. Kosinski, Y. Bachrach, G. Kasneci, J. Van-Gael, and T. Graepel,
‘‘Crowd IQ: Measuring the intelligence of crowdsourcing platforms,’’ in
Proc. 4th Annu. ACM Web Sci. Conf., 2012, pp. 151—160.
FEIJUAN HE received the B.S. degree from Xi’an
Jiaotong University, in 2010, and the M.A. degree
from The Hong Kong Polytechnic University,
in 2012, all in computer science. She is currently
an Associate Professor with the Department of
Computer Science, Xi’an Jiaotong University City
College. She has published more than 20 research
papers in various journals. Her main research inter-
ests include text mining, collective intelligence,
and e-learning.
YUDAI PAN received the B.E. degree in com-
puter science and technology from Xi’an Jiaotong
University, Shaanxi, China in 2015, and the M.S.
degree in computer science from Syracuse Univer-
sity, Syracuse, NY, USA. She is currently pursuing
the Ph.D. degree with the Faculty of Electronic
and Information Engineering, Xi’an Jiaotong Uni-
versity, Shaanxi. Her research interests include
machine learning and data mining.
QIKA LIN received the B.E. and M.E. degrees
from the Beijing Institute of Technology, Beijing,
China, in 2016 and 2019, respectively. He is cur-
rently pursuing the Ph.D. degree with the Faculty
of Electronic and Information Engineering, Xi’an
Jiaotong University, Shaanxi, China. His research
interests include deep learning and e-learning.
XIANGLIN MIAO received the B.S. degree from
Xi’an Jiaotong University, in 1976. is currently
a Professor with the Department of Computer
Science, Xi’an Jiaotong University City College.
He has published more than 40 research papers
in various journals and conferences. His main
research interests include data mining and infor-
mation management.
ZHOUGUO CHEN received the master’s degree
in signal and information processing from the
University of Electronic Science and Technol-
ogy of China (UESTC), in 2006. He is currently
a Network Security Senior Engineer with The
30th Research Institute of China Electronics Tech-
nology Group Corporation, Chengdu, China. His
research interests include network security, big
data, and network forensics.
170225