e-Picturebooks as Cognitive Artifacts [poster] (2016)

10th International Conference on Design Principles & Practices

e-Picturebooks as Cognitive Artifacts How storytelling is shaped by cognitive niche construction Thales Estefani, Pedro Atã, João Queiroz

Iconicity Research Group - Federal University of Juiz de Fora (UFJF), Brazil {thales.chaun, ata.pedro.1, queirozj}@gmail.com

PUC – Rio de Janeiro – Feb/2016

Here we introduce a theoretical framework for investigation of the cognitive and semiotic nature of digital storytelling. Our approach analyzes e-picturebooks (digital picturebooks) as cognitive artifacts and their impact on cognitive niche construction. Cognitive artifacts have the power both to create new problems and to create means for solving these problems. When both these aspects are taken into account, cognitive artifacts can be seen as shaping cognition itself: endowing it with both needs and capabilities, creating ever more specialized tools to deal with ever more specialized tasks.

Problem space: the possible causal links between semiotic entities (facts) and high level structures (narrative structures and themes, defined as boundary conditions or organizational principles) that influence these links Storytelling: the semiotic organization and communication of facts Cognitive artifacts: devices that allow navigation through the problem space (i.e., transition between problem states) reduce the cognitive cost of an operation (such as when using a calculator to perform a division)

Research questions and tentative answers 1 - How the problem space of storytelling is structured on cognitive artifacts? 2 - What are the specific semiotic features of e-picturebooks and how these features can alter storytelling production and interpretation? 3- How these alterations influence cognitive abilities regarding storytelling tasks? Answering question 1 requires an operational definition of storytelling as a problem space and identification of specific artifacts and semiotic features which produce effects that are observable and relevant for that problem space Answering question 2 requires analysis of examples and comparison between e-picturebooks and other forms of storytelling (digital or not) Answering question 3 requires a model of the integration of the semiotic features identified in the answer 2 and the inference of probable effects of these features in the storytelling niche. In the following paragraphs we present an initial plan for the investigation on the cognitive-semiotic nature of epicturebooks and some of its most salient features

Model proposal

Fundamental assumptions

increase its precision and efficiency (such as using a ruler to measure an object instead of just guessing its dimensions) allow new capabilities that would be impossible to the brain alone (such as using a graphical diagram to represent simultaneous relation between a large number of entities and infer specific visual patterns from it) participate in the creation of new problems and problem-spaces. (such as arabic numerals creating new possibilities for arithmetic calculation) Semiotic features: material qualitative properties of cognitive artifacts gestural repertoire superposition of non-linear interactive elements tendency of gamification navigation conditioned to a specific interaction multimodal textual forms multimedia resources (video, animation, audio) Cognitive abilities: examples include memory, perception, categorization, analogical inference, causal reasoning, abductive reasoning (hypothesis generation)...

A

problem space

cognitive artifacts

C

cognitive abilities

B A: cognitive artifacts participate in the creation of new problems and problem-spaces B: problem spaces function as boundary conditions which have a downward effect on the specific organization of cognitive artifacts C: cognitive artifacts may allow new capabilities impossible to the brain alone

Consequences A framework for analyzing digital storytelling from the co-evolution of problem space, cognitive artifacts and cognitive abilities.

Acknowledgements Pedro Atã and Thales Estefani would like to thank CAPES foundation. References: 1. Laland, K., Kendal, J.R., Brown, G.R.: The Niche Construction Perspective: Implications for Evolution and Human Behaviour. Journal of Evolutionary Psychology. 5 (1–4), 51–66 (2007) 2. Sterelny, K.: Thought in a Hostile World: The Evolution of Human Cognition. Wiley-Blackwell, Oxford (2003) 3. Queiroz, J. & Merrell, F. (2009). On Peirce´s pragmatic notion of semiosis – a contribution for the design of meaning machines. Minds & Machines 19: 129-143. 4. Ribeiro, S., Loula, A., Araújo,I., Gudwin, R. & Queiroz, J. (2007). Symbols are not uniquely human. Biosystems 90: 263-272.

Example analysis Jack and the beanstalk (Nosy Crow, 2014). Semiotic feature: superposition of nonlinear interactive elements. Before he/she continues reading, the reader can feed the cow, clean it with the brush and put a bell around its neck. This interferes on the visual characteristics of the story

Monster’s socks (Martin Hughes, 2015). Semiotic feature: navigation conditioned to specific interaction. Without pulling the bridge, the story cannot continue. Storytelling is conditioned to reader’s action

Rita, the lizard (Irene Blasco Grau, 2012). Semiotic feature: multimedia resources. Each element on the screen responds with a sound or animation when it is touched. It increases the engagement with illustration.

Numberlys (Moonbot Studios, 2012). Semiotic feature: tendency of gamification. In the story, the characters transform numbers into letters, but each transformation is a result of a simple task that the reader should perform. In this case, shooting at number 8. These tasks require a more direct participation of the reader in storytelling.