2018 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct)
RealityMedia: An Experimental Digital Book in WebXR
Maria Engberg* Jay David Bolter*
Dept. of Computer Science and Media School of Literature, Communication,
Technology, Malmö University and Culture, Georgia Tech
ABSTRACT
This paper presents an ongoing experiment using WebXR to
create something analogous to a non-fiction book in AR and VR;
an immersive, interactive experience that stands on its own, rather
than merely complementing a traditional book. The book
introduces the reader/user to AR and VR both as technologies and
as media. The printed book is one of the more influential
communicative interfaces in history. AR and VR have the
potential to remediate several genres of printed books, but
somewhat different conventions may need to be developed for
different combinations of genre and modality. The lessons learned
through this experiment should contribute to the establishment of
guidelines for this new form of multimedia, in particular
conventions that facilitate the reader/user’s transition from
discursive to immersive modes and back.
Keywords: Augmented reality, virtual reality, world wide web,
UX design.
Index Terms: • Human-centered computing~Mixed / augmented
reality • Human-centered computing~Virtual reality • Human-
centered computing~Interface design prototyping
1 INTRODUCTION
AR and VR are emerging as new media of communication and
expression, analogous to other media such as printed books and
films. Conventions for these new media are still in the early stages
of development. Some conventions may be generalized over all
possible uses and users, just as clicking on links became general
for hypertext on the original web, but others may vary with
content and application. Genres of AR and VR will likely
develop, just as there are genres of conventional websites and
other digital and non-digital media; the conventions in film (for
example) are different for documentaries, horror, fiction, drama
and so forth.
Exploring applications that have the potential to become
genres may aid in the development of such conventions. AR and
VR have the potential to refashion several genres of printed
books: including fiction, nonfiction, travel books, etc., and
appropriate conventions may need to be developed for each. Our
is to create an AR/VR version of a non-fiction book. It is work in
progress that we hope will ultimately suggest conventions for this
form of multimedia communication.
2 HISTORY OF THE DIGITAL BOOK
With the advent of the World Wide Web in the 1990s, the history
*maria.engberg@mau.se;jdbolter@gatech.edu; blair@gatech.edu
978-1-5386-7592-2/18/$31.00 ©2018 IEEE
DOI 10.1109/ISMAR-Adjunct.2018.00096
Authorized licensed use limited to: b-on: UNIVERSIDADE DE AVEIRO. Downloaded on March 13,2022 at 10:21:28 UTC from IEEE Xplore. Restrictions apply.
Blair MacIntyre*
Emerging Technologies, Mozilla
School of Interactive Computing, Georgia Tech
of publication and of the book reached a watershed. More and
more communication and publication moved to the new medium,
but at first the printed book still seemed secure as the medium for
the consumption of various forms of extended writing, such as
novels. In the first decade of the 2000s, however, ebooks emerged
as a serious rival to the printed book. At the same time “high-
concept” digital applications began to appear as tablet or
smartphone apps, e.g. Al Gore’s digital “book app” Our Choice or
The Fantastic Flying Books of Mr. Morris Lessmore (based on a
book by William Joyce). Such applications featured video and
audio along with graphics and text and suggested a new form of
the book that includes interactive elements for the reader to
explore.
AR and VR also began to be used to remediate the printed
book. The most influential example was and is the “magic
book”—a hybrid that combines the traditional printed codex
format with computer graphics that pop out of the pages. The
MagicBook was first developed by Billinghurst et al in the Hitlab
around 2000 [1]. It has subsequently become a genre. For
example, Marvel has produced a series of AR enhanced comics
that feature various forms of augmentation. Another recent
example is the recent Charlie Fink’s Metaverse [2], where pop-up
characters at the end of the chapters offer explanations of key
concepts of the text. This genre of augmented book is a
remediation of the children’s pop-up book of the 19 and 20 th th
centuries.
3 REALITYMEDIA: A DIGITAL BOOK
We report here on work-in-progress in the tradition of the digital
book: RealityMedia. In this experiment AR and VR constitute
both form and content. That is, RealityMedia uses VR and AR as
display technologies and the subject matter of the book is VR and
AR as technologies and as new media forms. The project spans
both traditional and digital technology. It consists of both printed
and digital versions. The printed version, whose work title is
Reality Media, will be published by MIT Press at the end of 2019.
It will situate AR and VR in historical contexts as media and will
also explain the technology and aesthetics of AR and VR. The
digital version RealityMedia is not meant to be a mere companion
to or illustration of the printed version; rather, it will function as a
stand-alone interactive experience of AR and VR. Our goal is to
join the two versions as closely as possible in material and
conceptual terms—to test the feasibility of a new kind of book,
realized simultaneously in print and in digital forms.
Understanding Comics
Inspiration for this project comes from Scott McCloud, the comic
book artist and author. In Understanding Comics [3] and
Reinventing Comics [4], McCloud offered an explanation of the
origin of comics, their nature today, and the future of this popular
media form. However, he did not produce his argument as a
traditional essay. McCloud came up with the ingenious idea of
324
writing his history and theory of comics in the form of a comic
book. He drew himself as a talking character, who delivers—
balloon by balloon—a long and fascinating lecture on the subject
to which he has devoted his life.
In a conventional book, the images serve as illustrations for
the text, but in comics images and text (e.g. speech balloons) are
fully integrated. They depend on each other to achieve meaning.
In addition, comics make meaning by combining image and text
across many panels in sequence. So, when in Understanding
Comics, McCloud defines how comics function as “sequential
art,” he can actually show the reader how panel sequences
construct narratives in a temporal flow. And he does so by
drawing himself as a character into the argument he is making.
McCloud’s thoughtful and at the same time playful approach
inspired us to try to explain a medium in and through the same
medium. What McCloud did for comics, we want to do for VR
and AR. However, there are important differences between our
project and his. One reason that McCloud was able to write a
treatise on comics in the form of a comic is that both a traditional
monograph and a comic exist in the same medium of ink on
paper. Both consist of text and illustrations, although in different
forms and ratios. In our case, however, AR and VR are born
digital media forms; they cannot function as reality media on the
printed page. It is difficult to illustrate AR and VR in the pages of
a printed book, and equally difficult to incorporate the same kind
of discursive text into a conventional AR or VR application.
So, while McCloud was able to express himself in a single
medium, we have to use two separate formats: print and digital.
These two formats enable us to present reality media from two
perspectives: from the “inside” as well as the “outside.” The
purpose of the digital version is not merely to illustrate the
affordances of the new media for the reader, but to allow her to
enact or even to inhabit the media. The printed book provides a
perspective that is the opposite of an immersive experience of AR
and VR: a critical explanation of these media in the contexts of
media history and their design aesthetics. The printed version can
also address the making of the digital version and reflect on the
future potential of the digital book. The printed book is
independent of the digital, although the two are in dialogue. The
reader will be able to understand the printed book without using
the digital version, but her understanding will be enhanced if she
experiences both forms.
There is another major difference between McCloud’s project
and ours. In Understanding Comics McCloud was writing about a
mature medium; he had the long and richly varied tradition of
comics to draw on for evidence. In our case AR and VR are just
beginning to constitute a medium, so we are speculating and
anticipating rather than describing a mature medium. VR and AR
are in the process of developing. In Gartner’s so-called Hype
Cycle of Emerging Technology [5], the two appear somewhere on
the “Slope of Enlightenment.” So we have to be speculative about
the future possibilities of VR and AR. Our speculations are,
however, based on the ways in which these technologies as new
media forms are remediating other existing media in the present.
This is always the way a new medium inserts itself into our
changing media culture. [6]
The portal as a conceptual metaphor
The portal is a key metaphor in our project, in three different
ways. First, the printed book is itself meant as a portal: an
introduction to the technologies and media experiences enabled by
AR and VR. Second, portals in the form of trackable images in the
printed book will allow the reader/user to move from the printed
325
Authorized licensed use limited to: b-on: UNIVERSIDADE DE AVEIRO. Downloaded on March 13,2022 at 10:21:28 UTC from IEEE Xplore. Restrictions apply.
version directly to relevant parts of the digital. This is in the
established tradition of the magic book, discussed above. Third,
the portal is an operative, visual metaphor in the digital version
itself for the relationship between AR and VR. In the digital
version the reader/user can find and cross portals, computer
graphic thresholds that enable her to move between the worlds of
AR and VR.
4 PRELIMINARY DESIGN SKETCH
The current, preliminary design envisions a coherent website
consisting of interlinked web pages on separate topics in AR and
VR. The reader can read the site on its own, but individual topics
will also be directly accessible from pages of the printed book
through the portal images mentioned above. Each page consists of
explanatory text, which parallels text in the printed book
(although these texts are less extensive than the ones in the printed
version). Embedded in each page will be one or more playable
AR/VR vignettes. The balance between (and manner of
presentation) text and 3D content will depend on the device the
user is using to access the content, and kind of “XR progressive
refinement” demonstrated by experiments at Mozilla [7] and
Google [8] over the past year. This page design with embedded
AR and VR is also beginning to appear on commercial websites.
The New York Times is already publishing articles that include
links to VR and AR demos—e.g. an article on an exhibition of
David Bowie’s costumes [9]. Our design is an elaboration of this
model.
Each top-level web page constitutes an immersive experience,
which may be experienced in various ways depending on the
device(s) the user has available. If the user has only a laptop or
desktop computer screen, then each vignette plays either
embedded in the page or in full-screen mode. The user uses the
mouse to interact. If the user has a headset attached to their
computer, then the immersive experience will open in the headset.
If the user has a tablet or smartphone, the experience makes use of
the interaction modes available (moving the device and touching
the screen). Finally, in a stand-alone AR or VR headset, the text-
based page will need to be presented in a manner appropriate for
the immersive context. Emerging AR/VR-first browsers allow
full WebXR pages to be viewed as 2D panes inside 3D. This is
one possible approach, but as we (and others) gain experience
with these devices, one of the design challenges will be how to
balance expository text with interactive content in such a purely
immersive experience.
Just as 2D web pages are connected through hyperlinks, the 3D
parts of the WebXR pages will be connected through visible
portals, as will the vignettes within a single page. The user may
travel from vignette to vignette before returning to the 2D
webpage to read text or find another 3D vignette. On a fully
immersive device, such as a head-mounted display, the user
returns to a top-level 3D space, e.g. a room. Figure 1 shows the
integration of a 2D textual page with the 3D vignettes in a sample
page in our current design.

Figure 1: The initial version of the “construct” vignette.
The text describes some of the basics of computer graphics. The
embedded VR vignette alludes to a scene in The Matrix: The
avatar welcomes the user to the “construct,” which is an
evacuated space in which the avatar and then the user can
construct and manipulate simple CG objects. This demo currently
works on a desktop computer and a tablet, and we are in the
process of extending it to other platforms.
5 WHY THE WEB AND WEBXR?
Ebooks are generally bought and distributed through the web, but
they are more often consumed on readers (such as the Kindle),
tablets or even smartphones rather than desktop or laptop
computers. RealityMedia too could have been developed as a
stand-alone app for the iOS and Android systems. But the advent
of WebXR technology has led us to choose the web as a platform
for a number of reasons.
The advantages of the web
There are well-known advantages to using the web for delivering
any kind of information to a general audience. Now that WebXR
is developing rapidly and promises to be available in major
browsers in the coming years, these advantages accrue to this
project as they would to almost any other web-based AR and VR
application. The web is by far the most widely available platform
and hence can reach the largest audience. Browsers exist for
virtually all computers, tablets, and smartphones, and should also
exist for the emerging stand-alone wearable AR/VR devices (e.g.,
the Oculus GO). The architecture of the web and the expectations
of its users mean that it is very easy to distribute updates (even
easier than with apps tied to the major app stores).
The ability to update has always been promoted as one of the
advantages of a digital book over a printed book, but with web
publication this advantage becomes a reality. It will make it
possible for us to correct text errors and programming bugs, to
improve existing vignettes, to add new text and vignettes
seamlessly, and to adapt to new devices as they become available.
The structure of web applications will allow us to experiment with
completely different approaches to presenting this content in a
326
Authorized licensed use limited to: b-on: UNIVERSIDADE DE AVEIRO. Downloaded on March 13,2022 at 10:21:28 UTC from IEEE Xplore. Restrictions apply.
relatively straightforward manner (i.e., different pages leveraging
the same backend content and services).
Finally, the web is appropriate for our project because it is
already an environment for the exchange of “serious” information,
including text. One stated original purpose of the World Wide
Web was to permit scientists and others to exchange drafts of
papers as well as data prior to official publication in print journals.
Of course, the web has grown to meet all sorts of other needs from
advertising and sales to gaming, but it remains one of the most
important environments for reading.
Immersion and critical distance
There is another reason we chose the web for this project, which
relates directly to the theme of the workshop: the issue of
immersion. The web became a multimedia platform with the
introduction of the <img> tag in the early 1990s. Audio and video
soon followed, although it took some time until increases in end-
user bandwidth permitted widespread use of these media. In some
of the most visited sites today, e.g. YouTube and Netflix,
photographs and video (and even 360° video) are the dominant
mode, whereas text is used for instructions, explanations, or users’
comments. In general, the web is both historically and today a
hybrid medium in which text and various forms of imagery and
audio are combined in various ratios.
This combination corresponds to the dual nature of
RealityMedia. The text of our books (in both the print and digital
forms) is discursive and analytical. It calls on the reader to reflect
on the historical, theoretical or technical significance of each facet
of AR and VR that is being described. The textual passages are
not intended to be immersive; they distance the reader from the
experience or technology under consideration. But when the
reader passes through the portal into either a VR and AR vignette,
the mode changes from discursive to immersive. WebXR allows
for the close integration of the interactive 3D content and the
textual material, which makes it possible to move back and forth
from the discursive/critical mode to an immersive mode.
The alternation between the modes of immersion and critical
distance is characteristic of genres of fiction as well as non-fiction
and of other media, including print and film. Some kinds of
novels and films try to engage the reader/viewer in an immersive
experience throughout, but other genres contain episodes in which
she is invited to reflect on the story in a critical way. Techniques
of critical distancing have therefore been developed for prose
(author’s asides, shifts from first person to third person, etc.) and
for film (direct address to the camera, “avant-garde” camera
techniques, etc.). This suggests that AR and VR as media can
develop analogous, though not necessarily identical, techniques
for narrative and descriptive experiences.
6 DESIGN ISSUES
In our prototype and sketches thus far, we are focusing on three
design issues: 1. support for multiple devices; 2. facilitating the
transition between critical/textual and immersive/participatory
modes; and 3. interactivity in the immersive mode.
Multiple devices
Because the goal is publication to the broadest possible audience,
a key design issue is the support of multiple devices with a variety
of display formats. Our aim is to provide the best experience to
the reader/user no matter what kind of device she is using, but we
cannot offer the same experience in all cases. If the reader/user is
viewing on a desktop computer than we cannot provide a truly
immersive VR experience and AR elements (that require an
opaque or see-through display, perhaps with a camera looking at
the world) become difficult or impossible.
Our current working solution is to design for a set of full-
experience platforms with diminished demonstrations for others,
following the common “progressive refinement” approach used
on current websites to adapt to phones, tablets and laptop/desktop
displays. One target platform is a laptop or desktop computer
screen for reading the textual material paired with an available
headset device that is capable of both VR and AR (i.e., a VR
device that includes an outward-facing camera, or a see-through
AR device that can be made opaque). Another target platform will
be tablets or smartphone devices capable of AR and VR (e.g. iOS
devices supporting ARKit or Android devices supporting ARCore
and Daydream), because these allow the easiest movement from
textual to immersive mode. As stand-alone AR and VR displays
become more common (e.g., Hololens, Magic Leap, Oculus GO,
HTC Vive, Lenovo Mirage, etc), we will explore the tradeoffs
between presenting the expository text (or variations of it)
immersively, or coupling those displays with tablets and
smartphones to create hybrid experiences. Time will tell what
approach is most satisfying and what users expect.
As far as web page layout is concerned, responsive frameworks
like Bootstrap (or similar frameworks like React) make it
relatively easy to accommodate any size and aspect ratio of the
screens of multiple devices. For the AR and VR immersive
experiences, the ability to adapt to multiple platforms is one of the
principal features of WebXR.
Transitions
As noted, the portal is our metaphor for transition, which we use
throughout the design. Portals provide transitions between the
printed book and the digital version and between the AR and VR
in the digital version. A key design issue is how to manage the
transitions between the text on the web page and the immersive
experience. Again, the techniques will depend on the capabilities
of the user’s platform. On the laptop-headset hybrid platform, the
user would have to put on the headset as she moves into the
experience and take it off again at the end. On the tablet or
smartphone, the transition is facilitated by the fact that the
WebXR is embedded in the web page, which would likely make
for a smoother reading experience (similar to the implementation
in the NY Times mobile app). A fully immersive presentation in a
stand-alone AR or VR headset may also yield a similarly smooth
experience. In this respect, the tablet and stand-alone displays
may come closest to the unified design of Scott McCloud’s
Understanding Comics, where, as we noted, McCloud achieved
the perfect integration of text and comic graphics.
Interactivity
One key advantage of a digital book over a printed book is that it
can involve the reader’s active participation in new ways. In
RealityMedia, we can include interactive vignettes so that the
reader becomes a user and learns by doing simple, scaffolded
experiments in a virtual environment. But one of the most
important current problems in AR and VR is how to enable the
user to interact in the virtual or virtual-physical space. We have
decided to design only for the simplest interaction modes initially,
using pointing, clicking, dragging to select and move objects, to
ensure the content can be accessible on the most platforms.
Furthermore, the digital book is meant to be an application with a
very broad audience, all of whom may not have the same
interactive skills as gamers or other special audiences. A book like
327
Authorized licensed use limited to: b-on: UNIVERSIDADE DE AVEIRO. Downloaded on March 13,2022 at 10:21:28 UTC from IEEE Xplore. Restrictions apply.
RealityMedia will not be a video game, and interaction modes
will have to be tailored accordingly.
7 CONCLUSION
RealityMedia is a work-in-progress. We are prototyping now and
hope to achieve a fully working version by the end of 2019, which
is when the printed book will appear. Our design is just one
possible way to remediate the printed book for AR and VR. It is
one small part of the larger task of media convergence, as AR and
VR are absorbed into our larger media economy in the coming
years.
The ultimate goal is to develop design guidelines for this genre
of book (long-form, non-fiction communication). Even at this
early stage, however, it seems likely that major design issues will
center on the relationship between the discursive (textual) and
immersive content.
RealityMedia is also an experiment in pushing the boundaries
of what it means to create web-based AR and VR experiences.
While experiences focused on specific devices or modalities (e.g.,
games, AR/VR education or training systems, etc) will likely
make up the bulk of initial uses of these technologies on the web,
we hope to demonstrate a more complex project that successfully
adapts to any web browser the user might use, and uses the
myriad of possible technologies to effectively engage the user.
REFERENCES
[1] Billinghurst, Mark, Hirokazu Kato, and Ivan Poupyrev.
“Magicbook: Transitioning between Reality and Virtuality.”
In CHI 2000, 25-26, 2001.
[2] Fink, Charlie. Charlie Fink’s Metaverse: An AR Enabled
Guide to VR and AR. Washington: Cool Blue Media, 2018.
[3] McCloud, Scott. Understanding Comics: The Invisible Art.
New York: HarperCollins, 1993.
[4] McCloud, Scott. Reinventing Comics. New York:
HarperCollins, 2000.
[5] Gartner, “Gartner Identifies Three Megatrends That Will
Drive Digital Business into the Next Decade,” Press Release:
August 15, 2017
https://www.gartner.com/newsroom/id/3784363
[6] Bolter, Jay David, and Richard Grusin. Remediation:
Understanding New Media. Cambridge, MA: MIT Press,
1999.
[7] Mozilla Mixed Reality Blog, “Progressive WebXR”,
https://blog.mozvr.com/progressive-webxr-ar-store/
[8] Google, “Article”, https://google-ar.github.io/WebAR-
Article/public/
[9] Ryzik, Melena. “Augmented Reality: David Bowie in Three
Dimension.” New York TImes (2018). Published
electronically March 20.
https://www.nytimes.com/interactive/2018/03/20/arts/design/
bowie-costumes-ar-3d-ul.html