Innovation in AR & VR

for Healthcare
July 2019
Presented by

xrdconf.com

AR/VR Healthcare | July 2019 1

Innovation in AR & VR for Healthcare
July 2019

W elcome to the second annual XRDC Healthcare Report,

a concise overview of how leading experts believe virtual reality,
augmented reality, and mixed reality technologies are shaping the state and
future of healthcare.

This topic is important enough that at XRDC 2019 there will be an entire track
of talks dedicated to how these technologies are revolutionizing medicine
by opening up new avenues of healing, improving the quality of care and
reducing the risks and costs of treatment.

To better understand the pace of progress we surveyed some top minds in

healthcare, technology, academia, and game development about how these
technologies are being used to augment treatments, improve the quality of
palliative care, assist in therapy, and more.

These insights were collected and presented by the organizers of XRDC

2019, which will take place October 14th and 15th at the beautiful Fort Mason
Festival Pavilion in San Francisco, a warm and convivial venue where leading
innovators, educators, and decision-makers can gather to trade insights and
build relationships.

XRDC 2019 will take place October 14th and 15th at the beautiful Fort Mason
Festival Pavilion in San Francisco, a warm and convivial venue where leading
innovators, educators, and decision-makers can gather to trade insights and
build relationships.

AR/VR Healthcare | July 2019 2

BLENDING ART AND SCIENCE
TO CREATE THERAPEUTIC
VR EXPERIENCES
Tell us about yourself and your recent work in AR/VR
After a chance encounter with a wild dolphin in 1993, followed by
over two decades of fascination with cetaceans, I founded the
Dolphin Swim Club in 2015 and was soon joined by my partner
Benno Brada as our producer.

The art project quickly grew into an organization that uses the
healing power of art and nature to craft therapeutic experiences in
virtual reality. The unique VR content is currently being used as an
evidence-based, drug-free medicine in over 500 hospitals and
healthcare institutions worldwide, and is present in several
scientific studies.
Marijke Sjollema,
Founder | Creative Director -
In 2017, the organization developed waterproof VR goggles that allow
The Dolphin Swim Club VR
users to swim in real water, with virtual wild dolphins, and experience
real therapeutic effects. As well as providing an entirely cruelty-free
alternative for so-called dolphin-assisted therapies with captive dolphins.

AR/VR Healthcare | July 2019 3

 What excites you most about AR/VR in 2019?
I stand in a long and rich Dutch tradition, as I was trained as a painter and still
wake up with that ‘eye’ every day. I imagine that artists like Rembrandt, van Gogh
or Mondrian, if they were alive today, would surely have embraced this entire
new and fantastic tech.

To create a world, and then be able to walk or swim or fly around in it; to create
a work of art that becomes a medicine; this is truly a new and unique path
available to me as an artist today, where the work you create can cross over from
being not just a product, but an experience to touch people’s lives with.

And of course, life will be easier with all kinds of improvements in the quality
of equipment, software, user friendliness, affordability etc. In this respect, I
think the future is looking bright.

Another exciting aspect of using our healing VR experiences to help patients

cope with pain or anxiety is that it leads to a reduction in the use of chemical
medicines. As we use lower doses of these drugs, not only do the patients
benefit, but it also contributes to cleaner water and a healthier environment.
So we all benefit.

What do you think is now the biggest challenge to realizing

AR/VR’s potential?
To connect talented medical scientists, artists, and technicians to collaborate
on amazing projects, so people in the dark hours of their lives can have the
best virtual medicine we can come up with. Also, finding the proper funding
and collaborators.

The Premier AR, VR & MR

Innovation Conference

■ Healthcare ■ Safety & Training

■ Games & Entertainment ■ Branding
■ Enterprise Applications

Register by August 21
to save $400
xrdconf.com AR/VR Healthcare | July 2019 4
There are already quite a few fantastic and very effective examples of what
magic can be achieved (and of course, I am proud that Dolphin Swim Club is one
of them). I like the example of the syringe; It must be sharp, it must be sterile,
but in the end what works is what you put in it.

Second, I think there is also a challenge to make an effort to provide some

assistance, and to get staff closest to the patients “on board” with this
technology to provide enthusiasm and support. All efforts are in vain if it all ends
up gathering dust on a shelf.

What’s one cool AR/VR application you think people

should know about?
I am a fan of VR/R experiences that offer freedom beyond the bounds of
traditional VR: location-based, immersive experiences that incorporating
more senses besides our sight and hearing. This is usually not possible or very
practical within healthcare.

So may I be a bit immodest to say that we are very proud

to have developed underwater VR goggles. As our skin
is our biggest organ, it is so immersive and a lot of fun Immersive tech is a powerful
to swim in real water with virtual wild dolphins. For this tool and we have seen some
technical achievement we received a Laval Virtual- ACM remarkable results. Not only
Siggraph Award, as well as an award from the French
in a pure healing sense, but
Polynesian Human Underwater Society.
in palliative care sense as
In a practical sense it is being used in aquatic therapies well, providing comfort and
with burn wound patients, revalidation, mentally or
peace of mind.
physically disabled swimmers during therapy sessions, and
animal-assisted therapy. To reduce pain, and bring joy and
motivation into exercises.

How do you work with the limitations of AR/VR to design effective

healing experiences?
We try to gather as much feedback from users and medical specialists as we can,
to learn what works, what their needs are, what limitations they have as patients
themselves. So if there is a limitation in movement, we focus on a specific field
of view without losing the immersive quality. We also consider the emotional
impact and the feelings of comfort someone might enjoy from having other
humans in the water, or whether they'd rather not. Or we play music or guided
meditations, if that's preferable, instead of natural sounds.

What works in VR/AR as medicine is very personal, so I believe strongly in

providing many options for what could bring effective healing. Not everything
works with everybody all the time, just like all other medicine. However if it
works, it works; immersive tech is a powerful tool and we have seen some
remarkable results. Not only in a pure healing sense, but in a palliative care sense
as well, providing comfort and peace of mind during the last weeks or days of
people’s lives.

AR/VR Healthcare | July 2019 5

ARTIFICIAL INTELLIGENCE
IS KEY TO NEXT-LEVEL
AR/VR EXPERIENCES
Tell us about yourself and your recent work in AR/VR
Hi, I’m Arno Hartholt, my background is in Computer Science and I’m
the Director of R&D at the USC Institute for Creative Technologies. I’m
responsible for combining our research capabilities into a common
development platform so that we can transition technologies to have an
impact on people’s everyday lives.

We often collaborate with the Army Research Lab to ensure that

military investments can benefit society as a whole. I’ve been working
with VR for over a decade now and it’s great to see how far it’s come
since then.

Most recently my team and I have focused on how best to develop Arno Hartholt,
virtual humans – interactive, AI-driven embodied digital characters – for Director of Research
a range of AR and VR platforms, including HoloLens, Magic Leap, and Development, USC
Vive, Oculus, iOS and Android. We’re also working with Dr. Skip Rizzo Institute for Creative
Technologies
to advance our virtual reality exposure therapy capability, BRAVEMIND,
a tool for clinicians to treat service members who suffer from
post-traumatic stress.

AR/VR Healthcare | July 2019 6

What excites you most about AR/VR in 2019?
The most exciting trend in this space right now is the convergence of ever-
improving hardware for end-users and ever-improving software for developers.
Combined, this results in a positive vicious cycle, where developers are able to
make more compelling applications more easily, which in turn allows end-users
to experience more engaging content more easily.

These reinforce each other, allowing both the audience as well as the
development pool to grow, which will ultimately lead to wide adoption of
these platforms.

What do you think is now the biggest challenge to realizing

AR/VR’s potential?
As a consumer product, a lot still has to happen: usability, price, fidelity and
content all have to improve. Having said that, I think the biggest potential will
only be realized once we can populate these virtual and augmented worlds with
actually intelligent and engaging virtual humans.

We need to go beyond mere personal assistants and create characters who

can act as mentors, role-players, tutors, etc. This not only requires fundamental
advances in artificial intelligence (AI), but also a lot of thought into the many
ethical and privacy related questions surrounding these technologies.

There’s a huge upside, though, not only for entertainment, but also for educating
and healing people and providing opportunities to constantly better ourselves.

What’s one cool AR/VR application you

think people should know about?
We created a virtual job interview training system for The biggest potential will only be
young adults with autism, called VITA, together with realized once we can populate
the Dan Marino Foundation and Magic Leap. these virtual and augmented
It allows students to practice a range of job interviews worlds with actually intelligent
with a variety of characters, who can act nice, neutral and engaging virtual humans.
or unfriendly. They see the character sitting on a
real-world chair right in front of them and they get
immediate feedback about say their level of eye
contact. It’s still early days, but the responses so far have been really
rewarding and encouraging.

How do you think the rise of wireless VR headsets will influence

your work?
Anything we can do to lower the barrier of entry for end-users is huge. Being
able to just pick up a piece of hardware, put it on, and get started is amazingly
helpful in actually getting people to use your application. While you typically
have less computing power available, it allows you to focus fully on designing a
core experience that’s user-friendly, engaging, and ultimately meaningful.

AR/VR Healthcare | July 2019 7

RESEARCHERS ARE NOW
COLLABORATING IN ALL-NEW WAYS
Tell us about yourself and your recent work in AR/VR
I've been at Merck for 6 years consistently working in areas focused
on or at least connected to edge-technology. Some examples are 3D
printing, IoT, wearables, XR, and RPA. Recently in the XR world I've
been working on re-imagining our onboarding experience. We're a
global company, so leveraging VR can allow people to get a lot more
perspective on the different parts of our company.

On a non-work level, I'm a director for Austin Virtual League, a non-

profit that gets VR into the hands of people who don't have the
opportunity, as well as hosting competitive VR LAN parties.

What excites you most about AR/VR in 2019? Zach Pinner,

For me it's a specific device for each area. For VR it's the Oculus Digital Technology &
Quest and it being cable-free, sensor-free, and computer-free. I've Solutions Lead, Merck
been enjoying mine a lot. There are definitely some drawbacks in
terms of power, but it's exciting to think about where this will lead.

For AR it's the Hololens 2. I haven't used one yet, but the advertisements I've
seen relating to the upgrades in how you interact look amazing.

What do you think is now the biggest challenge to realizing AR/

VR’s potential?
For websites and apps and other forms of technologies you have "best
practices" that have been learned and people can use when creating new
applications. A similar form of best practices doesn't exist for AR/VR. We're
still figuring out how to setup a UI and how to have people interact. How
controllers should be designed, or if they should be involved at all.

When you have a bad experience in AR or VR it can deter you from trying
it out again and considering a lot of people are taking their best guesses
at the user experience, you end up with a lot of really bad applications. I
think once some forms of UX best practices are realized, it'll be a big
challenge overcome.

What’s one cool AR/VR application you think people

should know about?
PLNAR on iPhone. Capturing the dimensions of my new place to buy furniture
was so easy. It has replaced my use for a tape measure in many cases.

AR/VR Healthcare | July 2019 8

How is AR/VR changing the nature of
drug research?
This new form of [virtual]
In drug research it allows for molecules to be viewed in new
perspectives and to collaborate over their structure in a brand
collaboration is what I
new interactive way without needing to be in the same room. think is the key value-add
This new form of collaboration is what I think is the key value- for AR/VR.
add for AR/VR.

How does your background in game dev influence

your work in AR/VR?
In game development you're focused on the player, you want them to be
caught in a flow state of enjoyment and challenge. I think this perspective fits
right into the user-centric change in design over the past decade.

When building something, I care about the user’s interactions with the tool
and I want there to be additional incentives to using the technology. More than
just getting the job done, it should be easy to use and even fun to use, maybe
even to the point of flow.

AR/VR Healthcare | July 2019 9

IMPROVING THE QUALITY
OF CHILDREN'S HEALTHCARE
WITH VR TREATMENTS
Tell us about yourself and your recent work in AR/VR
I’m a software developer turned VR designer turned entrepreneur. 15
years ago I was developing interactive projection systems, 3 years ago
I fell in love with VR, 12 months ago I started working in the medical
XR domain.

Recently we launched EmmaRye.com, a VR MRI simulator designed

to reduce anxiety and the need for sedation with young children and
patients with special needs.

What excites you most about AR/VR in 2019?

I expect the slew of new standalone headsets to open up the market
and make 2019 the turning point for mass adoption of VR. More users Shachar "Vice" Weis,
means more headsets and more content, which leads to greater Packet39.com
adoption of VR across all industries. Founder and CTO

AR/VR Healthcare | July 2019 10

What do you think is now the biggest challenge to realizing
AR/VR’s potential in healthcare?
Our biggest challenges so far have not been technical or medical, but
organizational. We have devoted a lot of resources to overcome red tape and
bureaucracy. Hospitals are organizations that move slowly by nature and they
are wary of cutting-edge technology.

Another interesting challenge is hygiene and

infection control, as most headsets were not Our biggest challenges so far have
designed with that in mind. We replace any fabric not been technical or medical,
or foam surfaces with non-porous 3D printed plastic
parts that can be wiped down and sterilized.
but organizational...hospitals are
organizations that move slowly
What’s one cool AR/VR application you by nature and they are wary of
think people should know about? cutting-edge technology.
Scanner Sombre is an obscure VR game, where you
explore a pitch black cave. The only way to figure
out your surroundings is to “scan” the walls with this
hand-held device that paints them with glowing particles. It’s not a new game,
but it’s unique and well worth a try.

What are the unique challenges of designing VR medical apps

for kids?
Small children pose a few interesting challenges. First, they can’t read, so
the VR experience has to be very intuitive and visual. Second, the audio and
animation styles need to be customizable to a degree; studies have shown
that boys and girls react differently to certain types of characters and voices.

Small heads sometimes need padding or custom straps.....

AR/VR Healthcare | July 2019 11

UNDERSTANDING THE NEED FOR
GOOD DESIGN IN MEDICAL-GRADE
VR EXPERIENCES
Tell us about yourself and your recent work in AR/VR
I'm a game designer and producer working with several companies
on games and health, many of which include some AR/VR work. I'm
currently SVP of Content for Mindmaze, a Swiss company that is doing
some exciting work in rehabilitation using games, VR, and Haptics.
Unfortunately my most interesting work in XR this year is on upcoming
products I can't yet talk about.

What excites you most about AR/VR in 2019?

I'm pleased to see that the tools for VR and AR keep improving, and that
there are both consumer products like Oculus Quest and more expensive
solutions with eye tracking and higher-res displays that will be good for
B2B applications. Noah Falstein,
SVP of Content,
What do you think is now the biggest challenge to realizing Mindmaze, SA
AR/VR’s potential in the healthcare industry?
The long testing cycle. Most game and enterprise software that works well
has lots of fast iterative development and testing loops.

Medical applications often need much more extensive, long-term tests

(particularly if you want FDA clearance) and that can be extremely expensive
and mean that the products may need to be implemented on the hardware
that was cutting edge 2 or 3 years earlier. There are exceptions, but some of
the most impactful innovations will require those long testing cycles.

What’s one cool AR/VR tool or application you think people

should know about?
I have been advising StoryUp, and their Healium VR/AR product is very
innovative and fun. They use the Muse headband for EEG sensing. Normally
consumer-quality EEG can be unreliable as your brow muscles can swamp
out the brain signal - but as Healium is focused on relaxation, they encourage
a positive feedback loop where the more the user relaxes those muscles, the
better they do - and the better the EEG signal becomes.

Coupled with a beautiful iPhone-based AR application, I think it's one of the

most creative, yet scientifically based relaxation therapies I've seen.

AR/VR Healthcare | July 2019 12

How has your experience in game development
influenced your AR/VR work, and your expectations
for the AR/VR market?
Working in the health/games overlap, I'm struck by how few companies
take the trouble to do both of those things very well. AR/VR is yet
another layer, and to get gameplay, AR/VR functionality, and the medical
components all working at top quality and complementing each other is
extremely challenging.

The key I've found is not forcing an ill-fitting game solution onto a medical
training or treatment option. When they are carefully aligned together, it
can be great, but it takes a lot of cross-disciplinary collaboration to make
it work. It reminds me of the work I did at LucasArts and Dreamworks,
getting people from the film and interactive industries to work together
and trust each other's unique expertise.

How did you learn to get people from those

two different industries to work together?
The short answer is, "trial and error". There were We need a future where
many interesting insights, from language (being "in people can just use their VR or
development" means very different things for games AR headsets in any area with
and movies) to paradigms (show, don't tell becomes do, minimal preparation. This year
don't show) to communications (at DreamWorks in 1995,
we're seeing some first steps
all the game people depended on email and rarely used
phones, Hollywood folks were the opposite). to making that a reality.

I learned the most from my close association with

[writer, director, and producer] Hal Barwood on Indiana
Jones and the Fate of Atlantis; we each taught each other a lot about our
respective fields.

What’s the most exciting AR/VR advance you’ve seen in the

last year? Why?
I'd say it's the availability of inside-out tracking headsets for consumers
becoming available at increasingly reasonable price points.

The process of dedicating a room in your house for VR and setting up base
stations is an interim solution, we need a future where people can just use
their VR or AR headsets in any area with minimal preparation. This year
we're seeing some first steps to making that a reality.

AR/VR Healthcare | July 2019 13