10th International e-Picturebooks as Cognitive Artifacts

Conference on How storytelling is shaped by cognitive niche construction

Design Principles &
Practices Thales Estefani, Pedro Atã, João Queiroz
PUC – Rio de Janeiro – Feb/2016
Iconicity Research Group - Federal University of Juiz de Fora (UFJF), Brazil
{thales.chaun, ata.pedro.1, queirozj}@gmail.com

Here we introduce a theoretical framework for investigation of the
cognitive and semiotic nature of digital storytelling. Our approach
analyzes e-picturebooks (digital picturebooks) as cognitive artifacts
and their impact on cognitive niche construction. Cognitive artifacts
have the power both to create new problems and to create means
for solving these problems. When both these aspects are taken
into account, cognitive artifacts can be seen as shaping cognition
itself: endowing it with both needs and capabilities, creating ever
more specialized tools to deal with ever more specialized tasks.
Research questions and tentative answers
1 - How the problem space of storytelling is structured on cognitive artifacts?
2 - What are the specific semiotic features of e-picturebooks and how these
features can alter storytelling production and interpretation?
3- How these alterations influence cognitive abilities regarding storytelling
tasks?
Fundamental assumptions Model proposal
Problem space: the possible causal links between semiotic entities (facts) and A
high level structures (narrative structures and themes, defined as boundary problem cognitive cognitive
conditions or organizational principles) that influence these links space artifacts abilities
C
B
Storytelling: the semiotic organization and communication of facts
Cognitive artifacts: devices that allow navigation through the problem space A: cognitive artifacts participate in the creation of new problems and
(i.e., transition between problem states) problem-spaces
B: problem spaces function as boundary conditions which have a downward
reduce the cognitive cost of an operation (such as when using a effect on the specific organization of cognitive artifacts
calculator to perform a division) C: cognitive artifacts may allow new capabilities impossible to the brain
alone
increase its precision and efficiency (such as using a ruler to measure an
object instead of just guessing its dimensions)
allow new capabilities that would be impossible to the brain alone (such Consequences
as using a graphical diagram to represent simultaneous relation between
a large number of entities and infer specific visual patterns from it)
A framework for analyzing digital storytelling from the co-evolution of
participate in the creation of new problems and problem-spaces. (such as problem space, cognitive artifacts and cognitive abilities.
arabic numerals creating new possibilities for arithmetic calculation)

Answering question 1 requires an operational definition of storytelling as a
problem space and identification of specific artifacts and semiotic features which
produce effects that are observable and relevant for that problem space
Answering question 2 requires analysis of examples and comparison between
e-picturebooks and other forms of storytelling (digital or not)
Answering question 3 requires a model of the integration of the semiotic
features identified in the answer 2 and the inference of probable effects of these
features in the storytelling niche. In the following paragraphs we present an
initial plan for the investigation on the cognitive-semiotic nature of e-
picturebooks and some of its most salient features
Semiotic features: material qualitative properties of cognitive artifacts
gestural repertoire
superposition of non-linear interactive elements
tendency of gamification
navigation conditioned to a specific interaction
multimodal textual forms
multimedia resources (video, animation, audio)
Cognitive abilities: examples include memory, perception, categorization,
analogical inference, causal reasoning, abductive reasoning (hypothesis
generation)...
Acknowledgements
Pedro Atã and Thales Estefani would like to thank CAPES foundation.
References:
1. Laland, K., Kendal, J.R., Brown, G.R.: The Niche Construction Perspective: Implications for
Evolution and Human Behaviour. Journal of Evolutionary Psychology. 5 (1–4), 51–66 (2007)
2. Sterelny, K.: Thought in a Hostile World: The Evolution of Human Cognition. Wiley-Blackwell,
Oxford (2003)
3. Queiroz, J. & Merrell, F. (2009). On Peirce´s pragmatic notion of semiosis – a contribution for the
design of meaning machines. Minds & Machines 19: 129-143.
4. Ribeiro, S., Loula, A., Araújo,I., Gudwin, R. & Queiroz, J. (2007). Symbols are not uniquely human.
Biosystems 90: 263-272.

Example analysis

Jack and the beanstalk Monster’s socks Rita, the lizard Numberlys (Moonbot
(Nosy Crow, 2014). (Martin Hughes, (Irene Blasco Grau, Studios, 2012).
Semiotic feature: 2015). Semiotic 2012). Semiotic Semiotic feature:
superposition of non- feature: navigation feature: multimedia tendency of
linear interactive conditioned to resources. Each gamification. In the
elements. Before specific interaction. element on the story, the characters
he/she continues Without pulling the screen responds transform numbers into
reading, the reader can bridge, the story with a sound or letters, but each
feed the cow, clean it cannot continue. animation when it is transformation is a
with the brush and put Storytelling is touched. It result of a simple task
a bell around its neck. conditioned to increases the that the reader should
This interferes on the reader’s action engagement with perform. In this case,
visual characteristics of illustration. shooting at number 8.
the story These tasks require a
more direct
participation of the
reader in storytelling.