A model for in-situ augmented reality
content creation based on storytelling and gamification
Fernando Vera, J. Alfredo Sánchez
Laboratory of Interactive and Cooperative Technologies
Universidad de las Américas Puebla
Puebla, México
{fernando.verapo, alfredo.sanchez}@udlap.mx
ABSTRACT
Augmented Reality (AR) has the potential to improve the user
experience in different contexts. This paper describes SituAR, a
model designed to facilitate the creation of stories with
gamification elements and AR content. Our approach on AR
content creation is based on storytelling elements to augment the
context in points of interest (POIs). We report preliminary work and
discuss the main components of the model such as social media,
storytelling, gamification, augmented reality and multimedia. Our
emphasis is on empower users to become authors in order to create
content in different scenarios. We are exploring new ways of
interaction to improve the user experience in POIs and promote
cultural heritage through stories. In this paper, the inception of
SituAR is discussed in addition to support scenarios and
preliminary prototypes.
CCS Concepts
Human-centered computing → Human computer interaction
(HCI). Computing methodologies → Mixed / augmented reality
Keywords
Augmented reality, authoring tool, user generated content,
storytelling.
1. INTRODUCTION
This paper presents progress on a model for Augmented Reality
content creation with a storytelling approach. We are interested in
using the social networks paradigm in order to create AR content
in a collaborative way.
The aim of the project is the exploration of AR authoring tools in
order to create a model where the user can create and interact with
augmented reality in-situ. Therefore, we have posed the following
research questions based on the current research topic and interests
of the scientific community.
Is it possible to develop an AR platform that allows users to
strengthen their social network through stories and cultural
information? Is it possible to improve the user experience in a city
with augmented reality? We are exploring new techniques for AR
content authoring in order to tell if it is possible to convert an AR
consumer to an AR prosumer.
There is a need to expand the context with information, we propose
situAR, a model to create interactive stories to add value in the user
experience at POIs in order to promote cultural heritage. It is
noteworthy that a variation of user-centered design is used as a
methodology in the research project.
Milgram and Kishino defined mixed reality as "anywhere between
the extrema of the virtuality continuum”, which gives users a way
to conceptualize the different mixes of realities in terms of how the
virtual and the real combine [9]. AR is defined as a live, direct or
indirect, view of a physical, real world environment whose
elements are augmented by computer-generated sensory input such
as sound, video, graphics or GPS data [4].
Nowadays, with the Web 2.0, website’s content from Facebook,
Twitter, YouTube and others, is constantly updated, since users are
continuously adding information to the social platforms. The
creation of augmented reality content can potentiate the integration
with social media. The user could create in-situ points of interest,
add new information to the real world from their experiences, and
share that information with other people. Augmented reality
browsers allow users to view the content added to reality, however,
the user interaction is limited by the fact that they may only provide
additional information about the context. With social media
elements, the user would stop being merely a consumer of
technology and would instead become a prosumer. Tentatively, this
means augmented reality would have greater social acceptance.
The remainder of the paper is organized as follows: Section 2
discuss related work; section 3 presents a study focused on the
context exploration; section 4 describes support scenarios based on
the previous study, section 5 introduces the conceptual design and
section 6 validates the model through preliminary prototypes.
2. RELATED WORK
This section justifies the importance of a platform for augmented
reality annotations. We based our approach on the notion of points
of interest (POI’s). One of our main concerns is how to represent
the information with AR markers and geo-location, in which [2] is
described as a floating label using GPS coordinates and it is one of
the most frequently used techniques. We can use geolocation and
markers to deploy augmented reality. Geolocation uses the GPS
signal of a mobile device; markers display content in real time [8].
QR codes are one kind of marker which allow the user to deploy
augmented reality content. The URL is stored in the code and we
can access that information with a mobile device in order to watch
or download associated content. It is important to use a mobile
device such as a smartphone with GPS technology in order to use
navigation maps in real time. In [1] it describes the process of
reaching a destination by overlaying arrows on the screen, which
shows the user the best route.
Recently there have been projects [10] that merge augmented
reality with social media. This fusion is called AR 2.0, and is a trend
because of the impact of social media.
We have heard about smart devices that are interconnected (Internet
of Things), and devices with augmented reality, such as Google
glass, which is a wearable computer with an optical head-mounted
display that is being developed by Google. But mobile devices such
as smartphones and tablets are the ultimate tools for the use of
augmented reality applications [5].
Hansen proposed a taxonomy (Figure 1) for annotations that
associates information with places [6].
Figure 1. Taxonomy for presenting annotations
Our approach is on attached annotations where the end-user and the
point of interest are co-located.
FitzGerald created a user generated content (UGC) for location-
based learning, where users actively create content and publish it
on the web and location awareness [3]. Väätäjä et al. explored AR
for user-generated hyperlocal news content to support and enrich
crowdsourcing, sharing and consuming of hyperlocal news content
created by the readers [12].
Langlotz et al. introduced the concept of Augmented Reality 2.0
(Figure 2). Their work describes the trend where users are
becoming content creators and not only consumers. This is one of
the main characteristics of the social web [7].
Figure 2. Augmented reality authoring tool
The Figure 2 describes actions that can be made into objects. In this
example, the application can copy an object such as a building and
paste it where the user wants to locate it.
Several efforts have been made to make augmented reality
approachable for the average user, e.g., AR games in mobile apps.
However, they don’t take advantage of social media elements and
folksonomies. Also, some AR applications are not focused on
exploring the user interaction at POIs.
Commercial applications are able to show POI’s nearby with
attached stories and can take users from one point or location to
another. Nevertheless, they are not connected with a social network
that can allow users to contribute comments, rate the places and add
multimedia elements of a POI’s such as videos, images or audio.
The main platforms focused on AR content creation are Sekai
camera, Layar, Wikitude, Junaio, Aurasma, Metaio and Google
Goggles (Figure 3). The limitations with these applications is that
the information is uploaded before use or they are desktop-based.
Figure 3. AR commercial apps
We found an opportunity to create an augmented reality platform
for points of interest enhanced to a social network in order to
generate information in real time. This proposal has elements of
social media, cloud computing and augmented reality. The idea is
to create dynamic content, and the user will feed the service by
using social networks. We explored cloud computer platforms such
as Windows azure and amazon services in order to access
multimedia elements from any location. Markers will be used to
connect with the application. There are services to create markers
such as Qodify, Daqri, Begrious and Grvoice. Markers can be used
with a QR code, a Microsoft tag or an augmented reality tag. In a
real scenario it will not be necessary to update the information of
the city because the users would be able to create annotations, add
content and share that information with other people.
The works described in the previous section have been developed
to create augmented reality content in the real world considering
different multimedia elements, such as text, audio, video or 3D
models, but those platforms do not take advantage of social media
by creating AR annotations. Also, most of the authoring tools are
addressed to developers, not an end user. We propose a model to
create in-situ AR content in order to promote social interaction and
cultural information using augmented reality technology. The goal
is to improve user experience through AR by involving the social
network to create AR content in a natural way for people without
programming skills.
3. CONTEXT EXPLORATION
A qualitative study was performed in order to understand the user’s
perception in a context with augmented reality. We applied surveys
to the users that focused on the way people interact with points of
interest (Figure 4). The purpose of the context exploration was to
understand the characteristics and problems that users discover on
points of interest and the degree level of the acceptance of
technologies such as mobile devices and augmented reality.
Figure 4. Interview with participants
Three potential users were interviewed to determine their interest
and the way they interact in the important places of a city; such as
museums and historical places. The purpose of the interviews was
to learn how people interact in POIs. Then, a scenario was created
with the use of technology, mainly augmented reality in points of
interest. Seven participants were evaluated, as a result of the
evaluation, the participants described their perception of the
concept, the elements missing, and their interest in AR content
creation (Figure 5).
• It helps to make • Add Games • Create pictures,
decisions • Interact with description,
• It needs content to learn location, reviews
Content Creation
conection with • Add maps to and ranking
social media navigate • Rewarding
Ideal Scenario
• It could have • Add near services (gamification)
problems with • Add interactive • Intuitive
gamification information in • Create narratives
Concept
• It is Interesting museums • Create videos in
but it have to be • Add stories historical places
easy to use.
Figure 5. Results of the scenario evaluation
The participants mentioned that AR technology in POIs is attractive
and engaging for tourists. It is important to add more information
in museums to promote cultural heritage. The participants of the
study showed interest to create content by sharing pictures, videos,
anecdotes and sharing their experience in historical places.
However, two of them were more excited to visualize content but
not to create it.
The information obtained was analyzed to understand the user’s
interaction in points of interest, such as museums and archeological
sites. Concept validation was necessary to detect elements for the
concept design.
4. SUPPORT SCENARIOS
We named our platform “SituAR”, which is a user-defined
augmented reality that integrates social media elements such as
comments, ranking and sharing. Also, it will allow the user to
perform some actions such as copy, paste, move, rotate, and scale
3D models. SituAR can be used in different contexts. Table 1
outlines the application of the different multimedia elements.
Table 1. Support scenarios of multimedia elements
This annotation system can be applied in different contexts. For
example, in real estate the architects could copy-paste a house to
show the client how it’s going to look when the project is finished.
In tourism, the users could interact with information related to the
context such as the story behind a painting, the visualization of a
video when the user is close to a place where a movie was filmed,
or taking of pictures with a 3D model that represents a specific
museum.
We created a storyboard in order to show the problems that people
have when they are in a specific POI. It also illustrates how the AR
annotations will work in-situ and off location.
The main problem of museums is the lack of information in objects
of interest. We are looking forward to have smart cities and
augmented reality technology; we can help to achieve that goal by
giving more information about the context within which the user
can interact. The storyboard highlights four challenges: How is the
user going to interact with AR technology? What kind of AR
content is going to be created by the user? How will the interaction
be off location? Which are the social media elements that the user
is familiar with? It is important to consider each of these elements
in the conceptual design of a platform for AR content creation.
5. SITUAR UNDERLYNG MODEL
We propose a conceptual design (Figure 6), which describes all the
components of SituAR and how the user interacts with them. The
model is based on content creation and visualization. The user can
create AR annotations in an open space in a natural way. We
defined two main features; Social and Multimedia elements. Taking
advantage of the social web, we included some actions that the user
is already familiar with. SituAR allows people to enrich the
platform with comments as well as rank comments and annotations.
Also, the user is able to share annotations with contacts and receive
notifications about nearby places or elements.
SituAR aims to create new ways of interaction by sharing and
designing experiences at POIs. The core of the model is the AR
content which the user could locate in different contexts. Moreover,
users will be able to share their knowledge creating stories. In order
to create a story, there are specific components in the storytelling.
The narrative and the characters are the core of the story. However,
the sequence component gives the structure to the story. The
gamification component allows the user to add game mechanics to
the story to enhance other users.
Social Media
Publish Stories
Share Narrative Augmented Reality annotation
Characters
Rank
Sequence Multimedia Objects
Comment Gamification
Figure 6. SituAR model
The user can visualize and locate multimedia objects in the
environment. Every multimedia element (audio, video, pictures)
will be represented by objects. For example, if a tourist wants to
share an old picture of a monument, the image will be represented
by a 3D object related with pictures. The AR annotations will be
symbolized with 3D objects in the location.
We propose a structure to create stories. Narrative, characters,
sequence and gamification allow users to tell a story including the
main characters of the story. Also, the sequence is needed to have
different locations and scenarios. Finally, we add the gamification
component where users can gain points when they create stories
and interact with others. These stories will use AR annotations to
provide a new way of interaction to the users. Moreover, SituAR
gives the tools to create the stories and AR annotations.
We have users at the top of the model represented by social media.
The users will be able to publish, share, rank, and comment stories.
SituAR will be the AR authoring tool that helps the user to design
new experiences and it proposes this elements:
SITUAR = {Users, Stories, Augmented Reality, Multimedia}
 Users = {Publish, share, rank, comment}
 Stories = {Narrative, Sequence, Characters,
[Gamification]}
 Augmented reality = {AR annotation, Multimedia}
 Multimedia = {Text, Audio, Video, Pictures, 3D
Objects}
6. VALIDATION BASED ON
PRELIMINARY PROTOTYPES
In order to validate the situAR model, we created a high-end
prototype which includes: User content creation and the
visualization of two different scenarios (“Hacienda UDLAP” and
“La Catrina”).
The goal of the prototype is to represent the components of the
model. The prototype includes options to upload pictures, videos,
audio or a story related with the context. These elements will be
geo-located so people could interact with the stories in-situ. The
users would be able to visualize AR nearby.
The prototype starts with the login interface. The user can sign up,
then log in into the application. The main interface (Home) is
represented with the information that your contacts have shared.
Also, it shows the stories nearby in order to interact with them. The
map shows the nearest stories considering the user’s location. The
profile has the information related with the user such as number of
stories published, contacts, among others. There is a gamification
section where users can see the trophies they have achieved.
Figure 7. SituAR home page
The menu has a story section where the user is able to create stories
by placing 3D objects in the environment. It is divided by scenes
where the user can add dialogs to the characters. In these scenes the
user can add different items in order to create a gamified story.
Figure 8. SituAR content creation interface
In order to engage people with SituAR, users are able to earn points
when they create a story.
7. CONCLUSION AND ONGOING WORK
SituAR proposes a platform to create augmented reality content. In
this platform, we involve the user to be the author of in-situ stories.
Storytelling and gamification give an important value to the
platform’s content. Mainly, because visitors can interact in an
immersive way with the POIs of a city. They can share their
experiences and knowledge while they learn more about their
context through interactive stories. At the beginning, we focused
our research on the AR authoring. Nevertheless, all the information
that the users create should be useful to the platform to build stories.
The use of augmented reality markers it has been contemplated to
interact in small spaces. The use of content based location will be
for open spaces. The AR content created by the user using the
platform is called AR annotation. This annotation can be a
multimedia element such as text or the merge of different types of
multimedia.
SituAR encourage users to visualize and create stories in-situ and
with mobile devices. This will allow visitors to learn more about
their surroundings and foment exercise.
As future work, the prototype will be evaluated with potential users.
The prototypes will have all the functions described on the SituAR
model. We will make a redesign of the prototype based on the
results of the user experience evaluation and we will continue
exploring new ways of interaction and an intuitive way to create
AR content. In the near future, users can interact with SituAR using
different mobile devices such as smartphones, tablets or AR glasses
in order to see the digital elements in the real world.
8. REFERENCES
[1] K. Akaho, T. Nakagawa, Y. Yamaguchi, K. Kawai, H. Kato,
and N. Shogo, (2011). “A study and evaluation on route
guidance of a car navigation system based on augmented
reality,” Proceeding HCII’11 Proceedings of the 14th
international conference on Human-computer interaction:
towards mobile and intelligent interaction environments -
Volume Part III, pp. 357–366.
[2] Choi, J., Jang, B., and Kim, G. J. 2011. Organizing and
presenting geospatial tags in location-based augmented reality.
Personal and Ubiquitous Computing, vol. 15, no. 6, pp. 641–
647.
[3] FitzGerald, E. 2012, Creating user-generated content for
location-based learning: an authoring framework. Journal of
Computer Assisted Learning.
[4] Graham, M., Zook M. and Boulton, A. 2012. Augmented reality
in urban places: contested content and the duplicity of code.
Transactions of the Institute of British Geographers, DOI.
[5] Gervautz M. and Schmalstieg, D. 2012. Anywhere Interfaces
Using Handheld Augmented Reality. Computer, pp. 26–31.
[6] Hansen, F.A. 2006. Ubiquitous annotation systems. In
proceedings of seventh conference on hypertext and
hypermedia, HYPERTEXT ’06 page 121, New York, New
York, USA. ACM Press.
[7] Langlotz, T., Mooslechner, S., Zollmann, S., Degendorfer, C.,
Reitmayr, G. and Schmalstieg, D. 2012. Sketching up the
world: in situ authoring for mobile augmented reality. Personal
and ubiquitous computing, 16(6), 623-630.
[8] Madden, L. (2011). Professional Augmented Reality Browsers
for Smartphones. John Wiley & Sons, p. 360.
[9] Milgram, P. and Kishino, A. F. 1994. Taxonomy of Mixed
Reality Visual Displays. IEICE Transactions on Information
and Systems. pp. 1321–1329.
[10] Schmalstieg, D., Langlotz, T., & Billinghurst, M. 2011.
Augmented Reality 2.0. In Virtual Realities (pp. 13-37).
Springer Vienna.
[11] Väätäjä, H.K., Ahvenainen, M.J., Jaakola, M.S, and Olsson,
T. D. 2013. Exploring augmented reality for user-generated
hyperlocal news content. In CHI'13 Extended Abstracts on
Human Factors in Computing Systems (pp. 967-972). ACM.