Startups
Augmented Reality in Reality
T
Haibin Ling he study of augmented reality (AR) can
HiScene be traced back to Ivan Sutherland’s inven-
tion of the head-mounted display (HMD)
in the 1960s,1,2 though it wasn’t until 1990
that Tom Caudell, a former Boeing researcher,
coined the term.3 Furthermore, for many years,
AR was confined to academia and sci-fi films.
Early AR systems were mostly experimental
and focused on specific tasks, such as main-
tenance and repair.4 More recently, AR has
started to successfully step away from laborato-
ries and appear in a wide range of applications,
including ones for advanced driver-assistance
systems, advertising, education, entertainment,
defense, manufacturing, medicine, smart cities,
social networking, and tourism.
In particular, the 2013 launch of Google
Glass was when AR first caught the attention
of a more general audience. Despite reluc-
tance from mainstream consumers, Google
Glass ignited a new enthusiasm for AR from
certain industries and techies. Since then,
tech giants have been actively developing
and investing in AR technologies (see the
“Investing in AR Hardware” sidebar), and the
progress is by no means limited to hardware
innovation.
AR on smartphones has been soaring
to new heights thanks to breakthroughs in
AR algorithms. In July 2016, Niantic and
Editor: Tao Mei, Microsoft Research Asia
Nintendo released Pokemon Go (Figure 1a),
triggering millions of downloads in one
week.5 One month later, social media giant
Tencent organized the virtual Olympic torch
relay on smartphones (Figure 1b),6 encour-
aging 100 million to use AR techniques pro-
vided by HiScene. The trend became clearer
in 2017, with Snapchat’s release of World
Lenses7 (Figure 1c) and the popularity of
Meitu’s facial-up app (Figure 1d). These apps
each have hundreds of millions of active
users.
10 1070-986X/17/$33.00 ©2017 IEEE
This rise in AR use stems from four main
developments:
■■ the pervasiveness of low-cost visual sen-
sors, such as phone cameras, which created
the foundation for AR consumerism;
■■ progress in environmental perception
algorithms, such as visual simultaneous
localization and mapping (SLAM), which
has provided key guidance for fusing vir-
tual information and reality;
■■ advances in optics, which largely pushed
the development of consumer-level AR
displays; and
■■ the maturity of multimedia techniques,
which enriched the content and styles of
AR applications.
In considering these developments, here I
review contributions from academia driving
AR’s newfound commercial potential, and I
consider the trends being exploited by industry
to advance software and hardware development
and emerging applications. I then discuss how
startups hoping to leverage these advances can
compete against senior tech tycoons in today’s
ultracompetitive environment.
Breakthroughs of
Enabling Algorithms
Although industry efforts are driving AR’s recent
popularity, it’s important to note the pioneering
contributions of academia. It’s beyond this arti-
cle’s scope to elaborate on all related breakthrough
algorithms; instead, I sample a few important
ones from the past decade that have significantly
influenced today’s AR. A more comprehensive
survey of milestones in AR appears elsewhere.8
2D Recognition and Tracking
In early AR systems, recognition and track-
ing was mostly based on 2D binary markers
Published by the IEEE Computer Society
 Investing in AR Hardware

T he following tech giants are actively developing or

investing in AR technologies:
References
1.   G. Sloane, “Watch Out, Snap: Mark Zuckerberg Outlines
Facebook’s AR Ambitions,” Advertising Age, 18 Apr. 2017;
■■ Microsoft has released its Hololens head-mounted AR http://adage.com/article/digital/mark-zuckerberg
device (www.microsoft.com/en-us/hololens), -confident-facebook-lead-snapchat-ar/308724.
■■ Google has launched Tango (https://get.google. 2.  L. Matney, “Apple Enters the Augmented Reality Fray
com/tango), with ARKit for iOS,” TechCrunch, 5 June, 2017; https://
■■ Facebook recently announced an ambitious AR techcr unch.com/2017/06/05/a pple-enter s-the
plan,1 and -augmented-reality-fray-with-arkit-for-ios/.
■■ Apple just showed off its new ARKit as the “largest AR
platform in the world.”2

Meanwhile, startup companies form another force for AR

development. Many of them, such as Atheer (atheerair.
com), Daqri (daqri.com), Magic Leap (magicleap.com),
Meta (meta.com), and Osterhout Design Group (oster-
houtgroup.com), and my own company, HiScene (his-
cene.com), have presented or forecasted their own AR (1) (2) (3)
headsets. Numerous AR headsets have been presented to
public, three of which are shown in Figure A—a pioneer- FIGURE A. Examples of head-mounted AR devices: (1) the
pioneering Google Glass, (2) Microsoft’s Hololens, which
ing headset (Google Glass), a product from a tech giant
received the Red Dot Award 2016, and (3) HiScene’s HiAR
(Microsoft’s Hololens), and a counterpart from a startup
Glasses, which received the Red Dot Award 2017.
(HiScene’s HiAR Glasses).

(essentially a 2D extension of barcodes), which and Mapping (PTAM)10 framework has

are inconvenient (because markers must be
attached to a real scene) or inappropriate
(because they appear unnatural) for many sce-
narios. This issue was largely relieved when 2D
pictures (movie posters, for example) started
to replace binary markers, as demonstrated
in Daniel Wagner and his colleagues’ seminal
work.9
Visual SLAM
For years, breaking the 2D restriction was
a key challenge in AR because of limited
computing resources. Then, in 1997, George
Klein and David Murray developed a SLAM
system running in real time on a smart-
phone.10 Since then, their Parallel Tracking
influenced modern SLAM algorithms on
smartphones, including the recently pop-
ularized ORB-SLAM.11 Another line of
work toward effective SLAM or 3D recon-
struction is via the fusion of multimodal
sources,12,13 which is common in newly
developed AR devices.
Efficient Computer Vision
Algorithms
General computer vision algorithms were not
extensively adopted in AR until the recent
emergence of fast yet reasonably accurate solu-
tions, including binary local features,14 fast
general tracking algorithms,15 and efficient
pose estimation.16

 July–September 2017 11
Startups
(a) (b) (c)
FIGURE 1. Four representative mobile AR products: (a) Pokemon Go, (b) QQ-AR (for a virtual Olympic torch relay),
(c) Snapchat AR, and (d) Meitu AR (meitu.com).
Deep Learning for
Semantic Understanding
Deep learning’s recent jumpstart in computer
vision has mainly been driven by the AlexNet
network for image classification, which was
published in 2012.17 Now, deep-learning solu-
tions are beating the competition in most vision
problems, such as object detection, semantic
segmentation, face recognition, and visual track-
ing. Although deep-learning solutions still face
computational limitations, their rapid advance-
ment is putting them in a good position for
future use in in AR.
AR Trends
The algorithmic breakthroughs paid off, thanks
to aggressive industrial follow-up and huge
market potential. Digi-Capital is reporting
that AR consumer-market revenue could hit
US$108 billion by 2021.18 AR’s fast growth will
be both horizontal and vertical. Horizontally,
the AR market will broaden its use to numer-
ous application domains while exploiting the
wide spectrum of emerging techniques. Verti-
cally, in-depth exploration of AR techniques for
specific industries (such as manufacturing) will
greatly speed up advances in the field.
Here, I consider trends in AR develop-
ment from two different perspectives: first
from the perspective of techniques, and then
from the perspective of applications.
12 IEEE MultiMedia
(d)
Advancing Software and
Hardware Techniques
In the near future, I expect to see advances in
both software and hardware techniques. In
particular, the following areas are receiving a
significant amount of attention.
3D environment perception. Automatic 3D
perception of the environment around an AR
device (such as a smartphone or AR headset)
has long been at the heart of AR. Many exist-
ing systems still live on 2D planar patterns due
to the lack of practical 3D solutions. This issue
has been alleviated recently by multisensor
fusion or effective algorithms—or a combina-
tion of the two. Without a doubt, reliable 3D
perception will greatly broaden the territory of
AR and elevate relevant user experiences.
Semantic perception. The fast evolution
of vision-based semantic understanding
algorithms—such as those for face analysis and
scene parsing—has a natural outlet in AR, pro-
viding content-aware guidance to augment a live
visual stream. An early example can be found in
Google’s Translate app. Semantic information
has not been extensively used in AR mainly due
to the lack of practical vision-based semantic
inference algorithms. With this situation improv-
ing rapidly, it’s easy to forecast the popularity of
semantic information in future AR products.
www.computer.org/multimedia
Virtual-real fusion. Although numerous AR
headsets have surfaced recently, they are still far
from offering a perfect user experience. Issues
related to the field of view (FOV), comfort level,
and cost are expected to be mitigated in the
near future. Furthermore, breakthroughs in AR
headsets will facilitate AR’s role as next gener-
ation human-computer interface. In addition,
new AR hardware systems, such as projectors
and heads-up displays (HUDs), will continue to
progress toward consumer markets. On the soft-
ware side, breakthroughs such as eye-tracking
algorithms could further refine rendering systems.
Smart human-computer interaction. With
the advances of environment perception and vir-
tual-real fusion, smart and natural HCI needs
to catch up as an important upgrade to AR.
Hand-gesture recognition and speech recogni-
tion, and the fusion of the two, are likely to be
the most important techniques along these lines.
Expanding Application Areas
It is safe to say that AR’s territory is expanding
to include many other fields, and exciting AR
applications are in the works.
From strong to weak virtual-real fusion.
Traditionally, AR has focused heavily on
high-quality, strong virtual-real fusion, which
is built on top of an accurate geometric under-
standing of the input stream. For example, in
Figure 2a, a fluent integration of the virtual
mask with the real face could not be achieved
without precise and fast localization of facial
landmarks. In practice, such a requirement is
usually expensive but not always necessary.
In many scenarios, rough localization of
targets of interest is sufficient. For example,
when augmenting a surveillance video, once
a target’s face is located, it typically suffices to
show related information around the suspect
(see Figure 2b) or even at a fixed position (for
example, on the corner of the image). The weak
virtual-real fusion in such cases needs only the
rough location of the targets, and is much more
casual than that in traditional AR.
This kind of weak fusion not only largely
reduces the burden of accurate environment
 July–September 2017
(a) (b)
FIGURE 2. Examples of (a) strong virtual-real fusion and (b) weak
virtual-real fusion.
perception, but also allows strong filtering tech-
niques to alleviate the notorious jittering effect.
An important driving force for such weakened
AR is advancements in semantic perception
techniques, which enables reliable and efficient
acquisition of semantic information.
From general to domain-specific AR. With
AR’s fast penetration into many application
domains, deep integration of AR with spe-
cific domains might show stronger competing
power than general AR. For example, AR in
clinical intervention might be closely related to
medical image analysis, while AR for tourism
typically integrates GPS-based location analy-
sis and geographical information systems.
From technique- to application-driven
products. As an emerging technology, AR
has remained mystic to most potential users.
It’s often an AR expert, instead of a domain
expert, who initiates the development of an
AR product. With the fast spread of AR to
general audience, and given the maturity of
AR techniques, domain experts are expected
to be more familiar and comfortable with AR
than ever before, and thus they could eventu-
ally initiate or even lead the development of an
AR product.
Opportunities for Startups
As described, AR has recently stimulated a tre-
mendous amount of interest and effort from
industries, ranging from senior tech tycoons
13
Startups
like Microsoft to young unicorns like Magic
Leap. Company involvement varies, but here
I focus on those developing AR techniques in
particular—as opposed to using AR.
If you work for an AR startup, you’ve likely
been asked how you plan to survive, competing
against all those tech giants. Every startup has
its own strengths, expertise, resources, culture,
and focuses, which work together to form a
unique growing path. That said, some common
guidelines still exist.
Work around Technique Limitations
Every technology has its inadequacies; so does
AR. As AR techniques continue to evolve, so
do their limitations. For example, real-world
AR used to be restricted to tracking 2D binary
markers, but recent AR products typically work
well with natural image patterns such as post-
ers. While a big company can afford to work on
far-reaching, long-term problems, most start-
ups can’t. That said, startups can still stay up to
date on the latest AR techniques and can work
around their limitations.
A startup should know the limitations of
AR techniques, be able to predict how these
limitations will change in the near future, and
be willing to push the limits. Taking 3D object
tracking as an example, a startup should know
that this technology is still in its infancy—
despite tons of research papers on the topic—to
avoid making over-aggressive promises to
potential clients. In addition, the startup
should have a rough projection of when, and
to what degree, the technique will mature, to
help plan ahead. Finally, startups that require
3D object tracking should devote serious time
and effort to pushing the limits.
Use a To-Business (2B) Model
For an AR startup, it is much safer to select
the 2B rather than to-consumer (2C) busi-
ness model for two reasons. First, 2C relies
heavily on channels that reach high-capacity
users, and such channels are seldom available
to AR startups. Note that this doesn’t mean
a startup can’t contribute to a 2C product.
In fact, this can be accomplished via collab-
oration with channel owners via the 2B2C
14 IEEE MultiMedia
model. For example, HiScene contributed to
the Tencent QQ virtual Olympic torch relay
by providing the underlying AR tracking
algorithm.
Second, a high-quality user experience
is often crucial in 2C products, and current
techniques often need improvements. This is
particularly true for AR headsets, which are
still under active development for better FOV,
lighter weight, and longer battery life. Differ-
ent from most 2C scenarios, users of 2B appli-
cations are relatively easy to satisfy, because
they usually care more about the functional
enhancement, productivity, and efficiency over
certain user experiences. For example, when
a mechanical engineer wearing AR glasses is
repairing a specific part, he or she cares more
about the part-relevant information—that is,
the model specifics—than about the precise
location in the picture where the information
is presented. In other words, weak virtual-real
fusion is sufficient for such 2B applications.
Foster Deep Integration
with Specific Industries
Because AR is still in its early stages, find-
ing universal solutions for all applications is
extremely hard—if not impossible. Conse-
quently, deep integration with a specific indus-
try is more practical for a startup to build a solid
protective barrier. Furthermore, big companies
will find it more difficult to pursue such deep
integration because of their historical burdens.
W e’re witnessing the rise of AR thanks
to the technical advances in relevant
areas as well as an increase in demand. I fore-
see the emergence and maturity of a rich set
of enabling technologies and prosperity of
a wide spectrum of AR applications, provid-
ing tremendous opportunities for startups to
make significant contributions to this growing
field.
References
1. D. Schmalstieg and T. Höllerer, Aug-
mented Reality: Principles and Practice, 1st
ed., Addison-Wesley Professional, 2016.
www.computer.org/multimedia
 2. I.E. Sutherland, “A Head-Mounted Three
Dimensional Display,” Proc. Am. Feder-
ation of Information Processing Societies
(AFIPS), 1968, pp. 757–764.
3. K. Lee, “Augmented Reality in Education
and Training,” Techtrends: Linking Research
& Practice to Improve Learning, vol. 56,
no. 2, 2012, pp. 13–21.
4. S. Henderson and S. Feiner, “Augmented
Reality for Maintenance and Repair
(ARMAR),” 2017; http://graphics.cs
.columbia.edu/projects/armar.
5. B. Molina, “‘Pokémon Go’ Fastest Mobile
Game to 10M Downloads,” USA Today, 20
July 2016; www.usatoday.com/story/tech
/gaming/2016/07/20/pokemon-go-fastest
-mobile-game-10m-downloads/87338366.
6. “Tencent QQ Drives Popularity of AR
Among Internet Users in China on the
Occasion of 2016 Olympic Games,”
PR Newswire, 2016; www.prnewswire
.co.uk/news-releases/tencent-qq-drives
- p o p u l a r i t y - o f - a r - a m on g - i n t e r n e t
-users-in-china-on-the-occasion-of
-2016-olympic-games-588631091.html.
7. C. Newton, “Snapchat Adds World
Lenses to Further Its Push into Aug-
mented Reality,” The Verge, 18 Apr. 2017;
www.theverge.com/2017/4/18/15333130
/snapchat-wor ld-lenses-something
-new-for-facebook-to-copy.
8. C. Arth et al., The History of Mobile
Augmented Reality, tech. report ICG–
TR–2015-001, Graz University of Tech-
nology, Inst. for Computer Graphics and
Vision, May 2015.
9. D. Wagner et al., “Real-Time Detection
and Tracking for Augmented Reality on
Mobile Phones,” IEEE Trans. Visualiza-
tion and Computer Graphics, vol. 16, no. 3,
2010, pp. 355–368.
10. G. Klein and D. Murray, “Parallel Track-
ing and Mapping for Small AR Work-
spaces,” IEEE Int’l Symp. Mixed and
Augmented Reality (ISMAR), 2007;
doi:10.1109/ISMAR.2007.4538852.
11. R. Mur-Artal, J.M.M. Montiel, and J.D.
TardÓs, “ORB-SLAM: A Versatile and
Accurate Monocular SLAM System,”
 July–September 2017
IEEE Trans. Robotics, vol. 31, no. 5, 2015,
pp. 1147–1163.
12. M. Li, B.H. Kim, and A. Mourikis,
“Real-Time Motion Tracking on a Cell-
phone Using Inertial Sensing and a Roll-
ing-Shutter Camera,” IEEE Int’l Conf.
Robotics and Automation (ICRA), 2013,
pp. 4712–4719.
13. R.A. Newcombe et al., “Kinectfusion:
Real-Time Dense Surface Mapping
and Tracking,” IEEE Int’l Symp. Mixed
and Augmented Reality (ISMAR), 2011,
pp. 127–136.
14. E. Rublee et al., “ORB: An Efficient
Alternative to SIFT or SURF,” Proc. IEEE
Int’l Conf. Computer Vision (ICCV), 2011,
pp. 2564–2571.
15. V. Lepetit, F. Moreno-Noguer, and P.
Fua, “EPnP: An Accurate O(n) Solution
to the PnP Problem,” Int’l J. Computer
Vision, vol. 81, no. 2, 2009; doi:10.1007/
s11263-008-0152-6.
16. J.F. Henriques et al., “High-Speed Tracking
with Kernelized Correlation Filters,” IEEE
Trans. Pattern Analysis and Machine Intelli-
gence, vol. 37, no. 3, 2015, pp. 583–596.
17. A. Krizhevsky, I. Sutskever, and G.E. Hin-
ton, “ImageNet Classification with Deep
Convolutional Neural Networks,” Proc.
25th Int’l Conf. Neural Information Process-
ing Systems (NIPS), 2012, pp. 1097–1105.
18. “After Mixed Year, Mobile AR to
Drive $108 Billion VR/AR Market by
2021,” blog, Jan. 2017; www.digi-capital
.com/news/2017/01/after-mixed-year
-mobile-ar-to-drive-108-billion
-vrar-market-by-2021/#.WUh0P-3yucx.
Haibin Ling is a professor at the Temple Uni-
versity. He is currently on leave and working
as the chief scientist at HiScene Informa-
tion Technologies, a startup company that he
co-founded with Chunyuan Liao and Rongx-
ing Tang. Contact him at linghb@hiscene.com.
15