Hardware

Machine Learning
Robotics
Security and
Privacy

JULY 2021 www.computer.org

 IEEE COMPUTER SOCIETY JOBS BOARD

Evolving Career
Opportunities
Need Your Skills
Explore new options—upload your resume today

Changes in the marketplace shift demands for vital skills

www.computer.org/jobs and talent. The IEEE Computer Society Jobs Board is a
valuable resource tool to keep job seekers up to date on
the dynamic career opportunities offered by employers.
Take advantage of these special resources for job seekers:

JOB ALERTS TEMPLATES WEBINARS

CAREER RESUMES VIEWED

ADVICE BY TOP EMPLOYERS

No matter what your career level, the IEEE Computer

Society Jobs Board keeps you connected to
workplace trends and exciting career prospects.
 IEEE COMPUTER SOCIETY computer.org

STAFF
Editor Publications Portfolio Managers
Cathy Martin Carrie Clark, Kimberly Sperka

Publications Operations Project Specialist Publisher

Christine Anthony Robin Baldwin

Production & Design Artist Senior Advertising Coordinator

Carmen Flores-Garvey Debbie Sims

Circulation: ComputingEdge (ISSN 2469-7087) is published monthly by the IEEE Computer Society. IEEE Headquarters, Three Park Avenue, 17th
Floor, New York, NY 10016-5997; IEEE Computer Society Publications Office, 10662 Los Vaqueros Circle, Los Alamitos, CA 90720; voice +1 714 821 8380;
fax +1 714 821 4010; IEEE Computer Society Headquarters, 2001 L Street NW, Suite 700, Washington, DC 20036.
Postmaster: Send address changes to ComputingEdge-IEEE Membership Processing Dept., 445 Hoes Lane, Piscataway, NJ 08855. Periodicals Postage
Paid at New York, New York, and at additional mailing offices. Printed in USA.
Editorial: Unless otherwise stated, bylined articles, as well as product and service descriptions, reflect the author’s or firm’s opinion. Inclusion in
ComputingEdge does not necessarily constitute endorsement by the IEEE or the Computer Society. All submissions are subject to editing for style,
clarity, and space.
Reuse Rights and Reprint Permissions: Educational or personal use of this material is permitted without fee, provided such use: 1) is not made for
profit; 2) includes this notice and a full citation to the original work on the first page of the copy; and 3) does not imply IEEE endorsement of any third-
party products or services. Authors and their companies are permitted to post the accepted version of IEEE-copyrighted material on their own Web
servers without permission, provided that the IEEE copyright notice and a full citation to the original work appear on the first screen of the posted copy.
An accepted manuscript is a version which has been revised by the author to incorporate review suggestions, but not the published version with copy-
editing, proofreading, and formatting added by IEEE. For more information, please go to: http://www.ieee.org/publications_standards/publications
/rights/paperversionpolicy.html. Permission to reprint/republish this material for commercial, advertising, or promotional purposes or for creating new
collective works for resale or redistribution must be obtained from IEEE by writing to the IEEE Intellectual Property Rights Office, 445 Hoes Lane,
Piscataway, NJ 08854-4141 or pubs-permissions@ieee.org. Copyright © 2021 IEEE. All rights reserved.
Abstracting and Library Use: Abstracting is permitted with credit to the source. Libraries are permitted to photocopy for private use of patrons,
provided the per-copy fee indicated in the code at the bottom of the first page is paid through the Copyright Clearance Center, 222 Rosewood Drive,
Danvers, MA 01923.
Unsubscribe: If you no longer wish to receive this ComputingEdge mailing, please email IEEE Computer Society Customer Service at help@
computer.org and type “unsubscribe ComputingEdge” in your subject line.
IEEE prohibits discrimination, harassment, and bullying. For more information, visit www.ieee.org/web/aboutus/whatis/policies/p9-26.html.

IEEE Computer Society Magazine Editors in Chief

Computer IEEE Intelligent Systems IEEE Pervasive Computing

Jeff Voas, NIST V.S. Subrahmanian, Marc Langheinrich, Università
Dartmouth College della Svizzera italiana
Computing in Science
& Engineering IEEE Internet Computing IEEE Security & Privacy
Lorena A. Barba, George George Pallis, University Sean Peisert, Lawrence
Washington University of Cyprus Berkeley National
Laboratory and University
IEEE Annals of the History IEEE Micro of California, Davis
of Computing Lizy Kurian John, University
Gerardo Con Diaz, University of Texas at Austin IEEE Software
of California, Davis Ipek Ozkaya, Software
IEEE MultiMedia Engineering Institute
IEEE Computer Graphics Shu-Ching Chen, Florida
and Applications International University IT Professional
Torsten Möller, Irena Bojanova, NIST
Universität Wien

2469-7087/21 © 2021 IEEE Published by the IEEE Computer Society July 2021 1
JULY 2021 � VOLUME 7 � NUMBER 7

18
Learning From
38
Parallel and
44
Flexibility in
Prototypes Distributed Production Systems
Systems by Exploiting
Cyberphysical
Systems
 Hardware
8 Long Tail Hardware: Turning Device Concepts Into Viable
Low Volume Products
STEVE HODGES AND NICHOLAS CHEN

18 Learning From Prototypes

ZBIGNIEW STACHNIAK

Machine Learning
28 Monolithically Integrated RRAM- and CMOS-Based
In-Memory Computing Optimizations for Efficient
Deep Learning
SHIHUI YIN, YULHWA KIM, XU HAN, HUGH BARNABY, SHIMENG YU,
YANDONG LUO, WANGXIN HE, XIAOYU SUN, JAE-JOON KIM, AND
JAE-SUN SEO

38 Parallel and Distributed Systems

MANISH PARASHAR

Robotics
40 To Err is Human, to Forgive, AI
PHIL LAPLANTE AND BEN AMABA

44 Flexibility in Production Systems by Exploiting

Cyberphysical Systems
ARNDT LÜDER

Security and Privacy

49 Bert Hubert on the Domain Name System
GAVIN HENRY

52 A Cybersecurity Terminarch: Use It Before We Lose It

ERIC OSTERWEIL

Departments
4 Magazine Roundup
7 Editor’s Note: Designing Hardware—from Prototype to Product
58 Conference Calendar

Subscribe to ComputingEdge for free at

www.computer.org/computingedge.
 Magazine Roundup
T he IEEE Computer Society’s lineup of 12 peer-reviewed technical magazines covers cutting-edge topics rang-
ing from software design and computer graphics to Internet computing and security, from scientific appli-
cations and machine intelligence to visualization and microchip design. Here are highlights from recent issues.
Visualizing Logical
Correlation in Trace Data
for System Debugging
This article from the March 2021
issue of Computer describes a
mechanism that enables debug
engineers to extract and visual-
ize the logical correlation among
events and messages in system-
level trace data. It enables debug
engineers to focus only on trace
packets that are logically corre-
lated to the issue under debug.
Cloud-Native Repositories
for Big Scientific Data
Scientific data have traditionally
been distributed via downloads
from data server to local com-
puter. This way of working suf-
fers from limitations as scien-
tific datasets grow toward the
petabyte scale. A “cloud-native
data repository,” as defined in
this article from the March/April
2021 issue of Computing in Sci-
ence & Engineering, offers sev-
eral advantages over traditional
4 July 2021
data repositories: performance,
reliability, cost-effectiveness, col-
laboration, reproducibility, cre-
ativity, downstream impacts,
and access and inclusion. These
objectives motivate a set of best
practices for cloud-native data
repositories: analysis-ready data,
cloud-optimized (ARCO) formats,
and loose coupling with data-
proximate computing. The Pan-
geo Project has developed a pro-
totype implementation of these
principles using open-source sci-
entific Python tools. By providing
an ARCO data catalog together
with on-demand, scalable distrib-
uted computing, Pangeo enables
users to process big data at rates
exceeding 10 GB/s.
From Lapel Pins to Coffee
Cups: Links Between
Corporate and Material
Culture, Lessons From IBM
Archeologists, sociologists, and
cultural anthropologists analyze
objects to gain insights about
human behavior. Can historians
use the same approach to study
corporate cultures? The evidence
Published by the IEEE Computer Society
says yes. This article from the Jan-
uary–March 2021 issue of IEEE
Annals of the History of Com-
puting discusses what such arti-
facts can teach us about modern
companies. IBM is used as a case
study because it is both iconic and
emblematic of large-scale twenti-
eth-century business. It is a com-
pany that continues to operate
within a highly defined corporate
culture. The authors describe a
selection of material objects and
their role in IBM’s activities. To do
that, they situate the objects into
Edgar Schein’s model of organi-
zational culture, since his is one
of the most widely accepted con-
structs of corporate culture.
Exploring the Design Space
of Sankey Diagrams for the
Food-Energy-Water Nexus
In this article from the March/
April 2021 issue of IEEE Computer
Graphics and Applications, the
authors define a set of design
requirements relating to Sankey
diagrams for supporting food–
energy–water nexus understand-
ing. They propose the network
2469-7087/21 © 2021 IEEE
 embodied sectoral trajectory dia-
gram design, a visualization design
that incorporates a number of
characteristics from Sankey dia-
grams, treemaps, and graphs, to
improve the readability and mini-
mize the negative impact of edge
crossings that are common in tra-
ditional Sankey diagrams.
An Efficient Solution to
Detect Common Topologies in
Money Launderings Based on
Coupling and Connection
Every money-laundering case
has a unique structure in terms
of transactions. It is not suffi-
cient to detect suspicious behav-
ior by just following the proba-
bility theory, where usually the
thresholds are given by experts.
Since the crime of money launder-
ing is more prevalent and sophis-
ticated nowadays, it will increase
the complexity of the detection if
the accounts with personal infor-
mation are combined with the
form of the transaction topology.
Hence, the graph topology analy-
sis could be used for anti-money
laundering tools. This article from
the January/February 2021 issue
of IEEE Intelligent Systems pro-
poses eight common topologies
based on coupling and connec-
tion from simple to much more
www.computer.org/computingedge
complicated structures to solve
various kinds of problems con-
cerning money laundering in the
real world.
HateClassify: A Service
Framework for Hate Speech
Identification on Social Media
It is a challenge for existing
machine-learning approaches to
differentiate hateful content from
content that is merely offensive.
One reason for this low accuracy
in hate detection is that these
techniques treat hate classifica-
tion as a multiclass problem. In
this article from the January/
February 2021 issue of IEEE Inter-
net Computing, the authors pres-
ent hate identification on social
media as a multilabel problem.
They propose a CNN-based ser-
vice framework called HateClas-
sify for labeling social media con-
tent as hate speech, offensive, or
non-offensive. Results demon-
strate that the multiclass clas-
sification accuracy for the CNN-
based approaches is competitive
with and even higher than cer-
tain state-of-the-art classifiers.
The results show that by using
multilabel classification, instead
of multiclass classification, hate
speech detection is increased by
up to 20 percent.
The Xbox Series X
System Architecture
The Xbox Series X console,
released in November 2020, con-
tains a system on chip (SoC) cre-
ated in partnership with AMD. This
article from the March/April 2021
issue of IEEE Micro describes the
architecture, including the intel-
ligence for input processing, ren-
dering game graphics and audio,
managing storage, user services,
and security—all under a tiered
operating system.
Toward Content-Driven
Intelligent Authoring of
Mulsemedia Applications
Synchronization of sensory effects
with multimedia content is a non-
trivial and error-prone task that
can discourage authoring of mul-
semedia applications. Although
there are authoring tools that per-
form some automatic authoring
of sensory effect metadata, the
analysis techniques that they use
are not generally enough to iden-
tify complex components that
may be related to sensory effects.
In this article from the January–
March 2021 issue of IEEE Multi-
Media, the authors present a new
5
MAGAZINE ROUNDUP
method that allows for the semi-
automatic definition of sensory
effects in an authoring tool. They
outline a software component to
be integrated into authoring tools
that uses content analysis assis-
tance to indicate moments of sen-
sory effects activation, according
to author preferences. The pro-
posed method was implemented
in the STEVE 2.0 authoring tool,
and an evaluation was performed
to assess the precision of the gen-
erated sensory effects in compari-
son with human authoring.
Smartphone Health
Biomarkers: Positive
Unlabeled Learning of In-the-
Wild Contexts
The DARPA-funded Warfighter
Analytics for Smartphone Health-
care (WASH) project is explor-
ing passive assessment methods
using smartphone biomarkers and
context-specific tests. The envi-
sioned context-specific assess-
ments require accurate recogni-
tion of specific smartphone user
contexts. Existing context data-
sets were either scripted or in
the wild. Scripted datasets have
accurate context labels, but user
behaviors are not realistic. In-the-
wild datasets have realistic user
behaviors but often have wrong or
missing labels. The authors of this
article from the January–March
2021 issue of IEEE Pervasive Com-
puting introduce a novel coinci-
dent data-gathering study design
in which data were gathered for
6 ComputingEdge
the same contexts using both a
scripted and in-the-wild study.
They then propose positive unla-
beled context learning (PUCL),
a method to transfer knowledge
from highly accurate labels of the
scripted dataset to the less-accu-
rate in-the-wild dataset.
A Systems Approach Toward
Addressing Anonymous
Abuses: Technical and
Policy Considerations
Can we prevent the abuses of
anonymous communication net-
works without affecting their abil-
ity to enhance privacy and evade
censorship? The authors of this
article from the March/April 2021
issue of IEEE Security & Privacy
evaluate approaches for balanc-
ing the need for anonymity with
the desire to mitigate anonymous
abuses.
How Trans-Inclusive
Are Hackathons?
Hackathons can be fun! However,
for the transgender community
and other minorities, hackathons
can have an uncomfortable atmo-
sphere. In this article from the
March/April 2021 issue of IEEE Soft-
ware, the authors present a survey
of 44 LGBTQIA+ people who are
experienced in hackathons and
interviewed seven transgender
participants. The authors intro-
duce five recommendations to
make hackathons more inclusive.
Influence of Technological
Resources on the
Development of
Mathematical Competence
in High School
The use of information and com-
munication technologies in the
process of learning through
technological resources can be
strongly motivating for students.
The use of appropriate digital con-
tent through simulations and ani-
mations can facilitate the under-
standing of complex content. In
this article from the March/April
2021 issue of IT Professional, the
incidence of the use of techno-
logical devices by students of the
mathematics subject in Second-
ary Education in Madrid (Spain) is
presented. The sample was of 31
students with a low level of com-
petence in the subject. A quasi-
experimental method of pre-test/
post-test design was used through
an experimental group and a con-
trol group.
Join the IEEE
Computer
Society
computer.org/join
July 2021
 Editor’s Note
Designing Hardware—
from Prototype to Product
H ardware is in high demand.
Today’s computing land-
scape, which includes mobile com-
puting and the Internet of Things,
requires many types of devices
that are optimized for diverse func-
tions—and answers could lie in
innovative hardware design. Large
corporations still produce most
computing hardware, but smaller
organizations and individuals are
entering the industry, thanks to
increased accessibility of technol-
ogies like 3D printing and off-the-
shelf development boards. This
ComputingEdge issue explores the
hardware-development process
and how new designs are further-
ing computer engineering.
In “Long Tail Hardware: Turn-
ing Device Concepts Into Viable
Low Volume Products,” from IEEE
Pervasive Computing, the authors
discuss accessible hardware
modeling tools, suggest ways to
2469-7087/21 © 2021 IEEE
overcome the challenges of bring-
ing a working prototype to mar-
ket, and present their vision for
a diverse hardware ecosystem.
In IEEE Annals of the History of
Computing’s “Learning From Pro-
totypes,” the author posits that
examining prototypes of early
computers can shed light on the
history of hardware development
and the electronics industry.
Hardware is driving many tech-
nological advancements, includ-
ing in machine learning (ML).
IEEE Micro’s “Monolithically Inte-
grated RRAM- and CMOS-Based
In-Memory Computing Optimiza-
tions for Efficient Deep Learning”
proposes techniques for energy-
efficient ML. Computer’s “Par-
allel and Distributed Systems”
describes a solution for accelerat-
ing training on GPUs.
Hardware is critical to
cyberphysical and robotic
Published by the IEEE Computer Society
systems—from driverless cars
to smart factories. The authors
of IT Professional’s “To Err is
Human, to Forgive, AI” ponder
how much trust we should place
in robots and intelligent systems.
Computer’s “Flexibility in Produc-
tion Systems by Exploiting Cyber-
physical Systems” details the
state of the art in Industry 4.0
components.
This ComputingEdge issue
concludes with two articles
related to online security and pri-
vacy. IEEE Software’s “Bert Hubert
on the Domain Name System”
considers the security and pri-
vacy implications of the Internet’s
Domain Name System (DNS). In
IEEE Security & Privacy’s “A Cyber-
security Terminarch: Use It Before
We Lose It,” the author argues
that the DNS Security Extensions
(DNSSEC) verification protections
are underutilized.
July 2021 7
 DEPARTMENT: SPECIAL ISSUE SPOTLIGHT
Long Tail Hardware:
Turning Device Concepts Into
Viable Low Volume Products
Steve Hodges and Nicholas Chen, Microsoft Research
A
t a time when the technology industry is
embracing powerful new algorithms and
cloud computing, continued innovation in
hardware is essential. In addition to the growing stor-
age and computation requirements of data centers
and edge computers, hardware devices provide the
critical gateway through which our systems receive
input and provide output. Whether it is intentional
user interaction, continuous context sensing, situated
information display, environmental monitoring, or
industrial control, we are more dependent on inter-
active and embedded hardware products than ever.
Indeed, as the Internet of Things grows, many predict
a dramatic growth in both the number and type of
such devices. A key factor in this growth is the ability
of those working in hardware to develop innovative
devices with new forms and functions.
Hardware development can be split into two
phases: first, a period of ideation, prototyping and
design iteration leads to new device concepts; and
then fruitful concepts transition beyond the basic
prototype. The latter phase typically involves creating
hundreds or thousands of copies of a prototype—
either preproduction evaluation samples or a fully
fledged low-volume product. The hardware device
research community and the industry it serves have
developed many tools and techniques to aid in the
ideation, prototyping, and design iteration phase
mentioned above. However, based on our first-hand
experience coupled with the observation of others,
we frequently see a bottleneck in the subsequent
phase—the transition from a working prototype to a
Digital Object Identifier 10.1109/MPRV.2019.2947966
Date of current version 21 January 2020
8 July 2021 Published by the IEEE Computer Society 
This article originally
appeared in
vol. 18, no. 4, 2019
viable product. The challenge of this bottleneck, its
root causes, and the potential benefits of overcoming
it are the subject of this column.
INTRODUCING THE LONG TAIL
Before examining the challenges of creating a via-
ble interactive or embedded hardware product, it is
insightful to understand how to characterize the suc-
cess of different products within a market. A common
approach is to plot a graph of the sales volume of each
different product in that market (on the y-axis) against
product rank (on the x-axis). This is called a rank fre-
quency distribution. Zipf's law 1 tells us that this dis-
tribution often follows an inverse power law, as illus-
trated in Figure 1. Such a distribution can be split into
two parts: the “long tail” to the right represents a large
number of niche (sometimes called boutique or cus-
tom) items, each of which is sold in relatively small
quantities; whereas the “head” at the left contains
just a few very popular items (the blockbusters), each
of which is sold in large volumes.
Market dynamics—factors like customer demand,
availability of supply, pricing, and competition—affect
not only the total quantity of products sold but also
the shape of the distribution. In some markets, the
tail is particularly heavy, meaning that in aggregate it
amounts to a lot of units. In other markets, the tail is
truncated. A good example of a truncated market is
the traditional movie industry. Here, there is a virtu-
ous cycle of positive feedback where established film
studios with prearranged global theatre distribution
produce blockbusters almost as a matter of course.
This generates large revenues to invest in future
productions that are also likely to be successful—a
process that economists call preferential attach-
ment. However, at the other end of the spectrum
2469-7087/21 © 2021 IEEE

would-be niche film producers struggle to find any
outlet for their films, creating a sharp cut-off in the
rank-frequency distribution.
TECHNOLOGY SHAPES THE TAIL
In microeconomic theory, it is recognized that tech-
nology affects the dynamics in a market.2 For exam-
ple, the growth of internet search and online shop-
ping has given rise to new ways for consumers to
discover and access products. In the case of down-
loadable digital products, such as e-books, music,
and movies, this has significantly changed the shape
of the rank frequency distribution. In these markets,
more of the niche products in the long tail are now sell-
ing because consumers can more easily discover and
obtain them. For example, Anderson describes how
the now-defunct movie rental company Blockbuster
historically reported that about 90% of in-store movie
rentals were new releases, whereas only 30% of Netf-
lix rentals were new releases because Netflix custom-
ers predominantly rent a large variety of back cata-
log movies.3 Similarly, there are more sales of long-tail
books online than in real stores. For example, Barnes
& Noble found that long-tail titles represented just 1.7
percent of its in-store sales, but the same titles took 10
percent of its online sales.3
Just as technological development has introduced
new product marketing and delivery mechanisms, it
has also eased product creation. In the case of movies,
the development of affordable digital cinematography
cameras and PC-based video editing have dramatically
lowered the barrier to entry for aspiring filmmakers.
Another example is the software market: since
the turn of the century, technological progress in
software development has had a dramatic effect
on the dynamics in this market. Development tools
have become easier to use and cheaper to obtain,
while agile, open-source development processes,
platforms, and communities that enable collabora-
tion have become firmly established. As a result, an
app can now be written by a relative newcomer to the
industry—perhaps someone who primarily wants to
address a personal need.
Another technological development that has
helped to enable many software products is cloud
computing. The ready availability of cloud services
removes the requirement for a company to make an
www.computer.org/computingedge
SPECIAL ISSUE SPOTLIGHT
FIGURE 1. Plotting a rank-frequency distribution often dem-
onstrate an inverse power law relationship with its character-
istic short head and long tail.1
up-front investment in server infrastructure, again
lowering the barrier of entry to newcomers and allow-
ing new ideas to be trialed quickly and easily. Many
cloud computing providers readily support dynamic
scaling and geographical redistribution should a soft-
ware product or service prove successful.
In the case of mobile software, all the technologies
on the previous page combine for even greater effect.
New products can be developed more quickly and
easily than ever before, the services that power them
can be deployed with little up-front expense, and the
“app store” marketing and distribution model provides
a channel to millions if not billions of potential users.
Even if a product only attracts hundreds of users a
year, it can still deliver useful revenue for the producer,
effectively adding weight to the long tail and increas-
ing customer choice along the way.
While the above examples of emerging long-tail
markets are all based on products that can be deliv-
ered digitally, it is worth noting that technological
developments have enabled a tail for physical goods
too.3 Amazon and Alibaba's e-commerce platforms
are the most visible examples of how marketplaces
that combine online search, a large customer base,
digital payment processing, and a fulfillment infra-
structure have changed the face of retail for physical
goods. Companies like Etsy take a similar approach to
enable sales of low volume boutique goods.
9
SPECIAL ISSUE SPOTLIGHT
HARDWARE DESIGN CONTINUES
TO GET EASIER
The last two decades of innovation in software design
tools, services, and processes have been mirrored in
the hardware industry. Powerful and well-documented
hardware design tools have been developed. These are
easy to use and install—and in some cases even run in
a web browser, e.g., Upverter. Entry cost has fallen and
fully featured design tools such as Autodesk's Fusion
360 are now available on a monthly subscription, while
others are free. At the same time, online communities
relating to design such as Electrical Engineering Stack
Exchange have become established, providing a way
to resolve design questions and troubleshoot issues.
The physical prototyping process for hardware has
also become more accessible. On the electronics side,
off-the-shelf boards from micro:bit to Raspberry Pi and
plug-together hardware platforms such as .NET Gad-
geteer,4 Seeed's Grove system, and MikroElektronika's
Click boards allow an initial prototype to be pulled
together in an afternoon. When more refinement is
needed, readily available ICs and electronic modules
increasingly abstract away underlying complexity.
Custom printed circuit boards (PCBs) are inexpensive
and quick to source, and companies such as Cir-
cuitHub and Seeed Studio provide an online service to
support the necessary PCB assembly. Sophisticated
hardware simulation and modeling tools are available,
and some—such as the LTSpice electronic circuit
simulator—are free. In terms of mechanical prototyp-
ing, improvements in rapid prototyping equipment and
services mean that laser cutting, three-dimensional
(3-D) printing, and computer-controlled machining are
less expensive and more readily available than ever.5
In addition, new technologies are continually
being introduced to further accelerate electronics
development. For example, the recently published
AutoFritz electronics computer-aided design system6
provides a kind of “auto complete” for electronic cir-
cuit design. It presents the designer with a list of sug-
gested components and circuit configurations based
on the software's analysis of circuits from thousands
of electronic component datasheets and existing
open-source electronic designs.
Collectively, these on-going innovations in the
hardware development space mean that nonexpert
individuals and organizations can prototype all
10 ComputingEdge
manner of interactive and embedded devices.5 Even
high-school students can design and build useful and
innovative hardware by combining an off-the-shelf
microcontroller board, some external components,
and a 3-D-printed enclosure. If current trends hold,
newcomers will be able to get started even more easily
and will be able to build more sophisticated devices,
while experienced developers will be able to work
more productively.
CROWDFUNDING AND
INCUBATORS MITIGATE
UP-FRONT COSTS
Of course, as mentioned in the introduction, the design
and prototyping phase is not the only challenge in cre-
ating a hardware product—a working prototype is just
the start. The next obstacle in hardware development
is the need for up-front capital investment in materials
and customized manufacturing equipment (tooling).
Another 21st-century phenomenon—crowdfund-
ing—helps to address this. Websites such as Kick-
starter and Indiegogo combine a prepayment mecha-
nism with a storefront akin to that of online retailers
and app stores. In this way, a hardware developer can
receive monies a year or more ahead of product deliv-
ery, easing the management of development costs and
mitigating the up-front costs associated with capital
equipment and raw materials.
Crowd Supply is a crowd-funding organization
that exclusively targets niche hardware products for
both amateur and professional electronics develop-
ers. As such, the profile of most of their customers is
similar to the profile of the product creators. Crowd
Supply maintains close involvement with each prod-
uct team: in addition to facilitating up-front payment
via its crowd funding portal, the company provides
know-how and experience, plus access to a network of
trusted partners to maximize the chances of success
for the projects it agrees to support.
Start-up incubators also provide hands-on support
and mentoring, often coupled with venture capital (VC)
investment—all in return for a stake in the start-up.
Shenzhen-based Hax and Silicon Valley's Lemnos Labs
and Highway 1 focus on hardware product incubation,
drawing on their hardware expertise and contacts to
minimize the chance of missteps. Hardware Club is a
community-based venture firm whose members share
July 2021

experience, knowledge, and contacts in order to move
more quickly while simultaneously reducing risk.
It is worth noting that incubators typically pursue
products with high growth potential—things that will
quickly move out of the long tail. Companies who want
to avoid the pressure that often follows VC invest-
ment and grow a new business more organically can
employ the services of an external design service or
consultancy in order to access product development
expertise. Companies such as Dragon Innovation and
PCH International specialize in enabling the prototype
to production transition for electronic products.
BUT HARDWARE IS STILL HARD
The collective benefits of online storefronts, accessi-
ble and powerful design tools, crowdfunding, incuba-
tors, consultancies, and next-day delivery networks
all reduce cost and mitigate risk for those seeking to
bring a new hardware product to market. But evidence
suggests that they are often not enough. The flip-side
of preferential attachment is that nascent devices
often face a vicious circle. To be successful they must
combine significant utility with compelling user expe-
rience, while simultaneously being robust and reli-
able in operation. Meeting these goals requires a large
design and engineering investment, both for the prod-
uct itself and for its manufacturing process—and that
is hard to justify before a product is successful. As a
result, there are many instances where new consumer
hardware products fail.
One example comes from Central Standard Tim-
ing, a two-person start-up that designed the world's
thinnest watch. Based on a sophisticated working pro-
totype, Central Standard Timing had tremendous suc-
cess on Kickstarter—they raised over a million dollars
from backers who paid either $99 or $129 for a watch. In
moving from prototype to product, they chose to part-
ner with one of the world's most experienced electron-
ics manufacturing partners. But ultimately, they were
unable to create a reliable production process—less
than ten thousand units was a small volume for their
manufacturer, and ultimately the per-unit cost of the
watch was rumored to be $260.9 As a result, Central
Standard Timing were not able to deliver the product
to their backers and ultimately ceased operations.
TriggerTrap, a company that manufactured devices
to allow SLR cameras to be triggered externally,
www.computer.org/computingedge
SPECIAL ISSUE SPOTLIGHT
provides a second example of the difficulty of moving
from prototype to product. TriggerTrap's first hard-
ware offering was based on an Arduino prototype
and resulted in a successful Kickstarter campaign.
However, they experienced difficulties sourcing
enough of the displays used in their prototype to meet
a production volume of nearly one thousand units.
The part was no longer available and TriggerTrap had
to convince the display manufacturer to instigate a
special low-volume production run before they could
fulfil their orders.10 TriggerTrap's second hardware
product—the Ada—also resulted in success on Kick-
starter, raising £290,000. However, the company was
unable to transition from their prototype and failed
to deliver any Ada products. Although they brought in
expertise from external consultancies, they still made
costly mistakes and the consultancy fees were an
additional drain on their resources.11
Central Standard Timing and TriggerTrap both
failed due to difficulties navigating the transition from
prototype to product. They leveraged many of the
mechanisms previously described—such as powerful
design tools, online storefronts, crowdfunding and
experienced professional design, and manufacturing
partners—but nonetheless they still underestimated
the cost and complexity of transitioning from a refined
prototype to a product that could be manufactured in
volumes of hundreds to thousands.
Of course, an experienced team with a track record
of hardware product development is already well
versed in the transition from prototype to product.
But even then, it is still possible to underestimate the
scale of the challenges involved. Our final example
comes from SenseCam, which started as a wearable
camera research project12 before being productized by
an established device manufacturer. Despite the com-
pany's experience with hardware manufacture, they
ultimately found the vicious circle too difficult to over-
come. The first and second versions of the SenseCam
product (the Revue and Autographer, respectively)
successfully sold in modest volumes to researchers,
clinicians, and enthusiasts. But as the product design
evolved on a trajectory toward a compelling and com-
petitively priced consumer product, it became clear
that an investment of tens of millions of dollars was
necessary to reach sufficient economies of scale.13
This level of investment was ultimately not justified
11
SPECIAL ISSUE SPOTLIGHT
given the risk inherent in a new product category such
as a wearable stills camera, and production ceased.
THE REPLICATION CHALLENGE
We believe that many of the difficulties in creating a
new hardware product, as encountered in the exam-
ples above, stem from a fundamental challenge. Unlike
digital products which can be replicated in a sim-
ple way requiring almost no resource and resulting in
perfect copies, the process of replicating a physical
device is complex and incurs cost, and no-matter how
much is spent on manufacturing, the copies are subtly
different. We call this “the replication challenge.”
Compared to a smartphone app that can be
released to an audience of millions of people when
development and feature testing are complete, scal-
ing a hardware device from “I have one that works” to
“anyone can buy one”—i.e., transitioning from a work-
ing prototype to a viable product—is much harder.
Although this difference between software and hard-
ware productization is easy to describe, the number
and complexity of steps required to take a device to
production is much harder to comprehend until expe-
rienced first-hand. Essential activities associated with
the replication challenge8 include:
›› finding reliable suppliers for all components and
materials;
›› accommodating component tolerances;
›› designing and building the necessary tooling;
›› building an efficient and reliable manufacturing
process;
›› controlling and accommodating manufacturing
variability;
›› selecting and managing manufacturing
partners;
›› instigating and maintaining manufacturing
quality control; and
›› adapting to changes in pricing and availability of
components and services.
These activities are by no means unique to elec-
tronic device production—they apply to the produc-
tion of nearly all physical products. However, they
are worth highlighting for two reasons: first, many
of them involve more complexity and cost for elec-
tronic devices than for other physical products; and
12 ComputingEdge
second, many of these activities are unfamiliar to
those steeped in the software-side of technology.
The replication challenge does not only apply
to high production volumes—in fact as volumes
increase beyond ten thousand units a year many
well-established manufacturing processes that lever-
age economies of scale become viable, so the cost of
the above activities is more easily absorbed. Instead,
replication is particularly challenging at lower volumes.
For consumer products, the difficulties typically start
at around one hundred pieces per year when the same
ad-hoc craft production techniques used to make
prototypes are no longer sufficient. A compounding
factor with low volume manufacturing is its natural
reliance on batch production, which introduces the
further challenge of periodically reinstigating the
entire manufacturing process and supply chain.
As a result, our sense is that it is increasingly rare
for low volume consumer devices to be viable in the
market. While Zipf's Law predicts a long tail of demand
for niche hardware, the economics of production con-
strain the economic viability of such products, and the
tail is truncated. As a result, consumer choice is limited
to a relatively small number of high-volume devices.
These are typically made by large companies who can
justify the sufficient resource required to bring them
to market and sustain them, and often only when they
amplify or otherwise complement an existing product
line.7 In this environment, the recent successes in the
interactive and embedded hardware space include
the well-known smartwatches and voice assistants
from major companies. These have created enough
momentum in the market to transition from vicious
to virtuous positive feedback and generate enough
revenue to warrant on-going refinement.
CHARACTERISTICS OF LONG TAIL
HARDWARE SUCCESSES
Of course, there are examples of commercially via-
ble low volume consumer devices. Armed with a bet-
ter understanding of what is necessary to transform a
hardware prototype into a low volume product—and
some of the pitfalls to look out for—it is interesting to
examine some of the characteristics of these success-
ful products.
Like SenseCam, Circuit Stickers14 started life
in research and quickly transitioned into a product
July 2021

through the efforts of Chibitronics, a small self-funded
start-up. From the outset, Chibitronics understood
the importance of reliable and cost-effective sup-
ply and manufacturing, recognizing that the design
complexity of manufacturing tooling can out-weigh
that of the product itself.15 They had realistic expecta-
tions of production costs and were able to fulfill their
crowd-funding obligations.
Subsequent to this, Chibitronics has successfully
established a small range of niche products that sit
in the long tail. In doing so, the company has started
to turn positive feedback to its advantage: it is now
able to amortize investments creating relation-
ships with suppliers, manufacturing partners, and
distributors, across multiple products. Although no
individual product is manufactured in huge numbers,
the aggregate volume across multiple products is
large enough to negotiate better pricing and attract
more commitment from component suppliers and
manufacturing partners.
Improving efficiency across multiple products, as
compared with the efficiency of manufacturing any
one in isolation, is termed “economy of scope.” While
economies of scale are characterized by volume,
economies of scope are efficiencies formed by variety.
The hardware crowd-funding specialist Crowd Supply
also leverages economies of scope by drawing on a
specific set of partners to manufacture many of the
products in its portfolio. In a similar vein, Crowd Sup-
ply takes advantage of its partnership with Mouser, a
large electronic component distributor, to secure reli-
able and competitively-priced supply.
A characteristic of Crowd Supply's wide range of
successful products is that many take the form of a
“bare board”—a PCB that has components soldered
to it but comes with no enclosure. This is perfect for
Crowd Supply's target audience of hardware devel-
opers and hobbyists. It also avoids the high up-front
cost often associated with designing and manufac-
turing tooling for an enclosure, which can be thou-
sands, if not tens of thousands, of dollars for each
plastic piece. In contrast, TriggerTrap, which needed
an enclosure because the company was targeting a
different audience, ran into difficulties transitioning
from the rapid prototyping used to create its pro-
totypes to manufacturing processes suitable for a
product. Ultimately, the company could not reconcile
www.computer.org/computingedge
SPECIAL ISSUE SPOTLIGHT
the tooling cost of injection molding with its relatively
low production volume.11
Central Standard Timing did not require injec-
tion molding for its product but faced a different
challenge. In order to deliver one of the thinnest
electronic devices ever produced, the company
needed to use nonstandard electronics assembly
techniques. Unlike PCB assembly, which is still cost
effective even when scaled down to low volumes, the
direct chip-to-flex process required costly tooling
and introduced nonstandard steps in the manufac-
turing process, making it error prone and unreliable.
Relying on leading-edge manufacturing technology
in this way is particularly risky for start-ups.13 In con-
trast, the established and well-understood manu-
facturing process for a standard PCB bare board
product results in few defects and can be readily
transferred between manufacturers, stimulating
healthy competition.
UNLOCKING THE LONG TAIL:
A CALL TO ACTION
Based on our analysis of some recent successes and
failures in taking new hardware concepts to market,
we do not believe that turning an idea for a new device
into a working prototype is a limiting factor in new
product introduction. Instead, we believe that the bot-
tleneck is the transition from a working prototype to a
viable product. Therefore, we encourage those work-
ing in the field of device hardware to join us in tack-
ling this transition. Collectively, we would like to work
toward three broad goals.
Improved Teaching Materials
and Education Programs
We need richer learning materials regarding the pro-
ductization process: new ways to share the knowledge
that today largely resides in the design consultancies
and large companies with the first-hand experience of
device manufacture. Fortunately, there are already a
few books that cover the topics raised in this paper;
we encourage those interested to read “The Hardware
Hacker” by Andrew ‘bunnie’ Huang,15 “Prototype to
Product: A Practical Guide for Getting to Market” by
Andrew Cohen8 and “IoT hardware from prototype to
production” by Richard Marshall, Lawrence Archard,
and Steve Hodges.16
13
SPECIAL ISSUE SPOTLIGHT
We would like to see these books complemented
by other learning materials. For example, curated
virtual tours of device manufacturing facilities would
provide valuable insights to those who cannot visit
in person. We also encourage universities to provide
more coverage of topics relating to the replication
challenge in undergraduate- and graduate-level level
courses and in professional development programs.
Stronger Communities and Tighter
Integration Between Partners
We see an opportunity to extend today's established
online communities and professional networks in a
way that allows newcomers to the device hardware
space to engage with each other and with established
players so that they can more easily form the part-
nerships that are vital to the delivery of a successful
product.
We imagine that these partnerships will increas-
ingly span the globe. In the short term, we would like
to streamline access to resources in today's leading
electronics manufacturing regions. For example, the
geographical proximity of capital equipment, skilled
and unskilled labor, and raw materials in locations
such as Shenzhen provides unrivaled economies of
scope. Indeed, much of the Shenzhen ecosystem
naturally lends itself to batch production: there are
thousands of small factories specializing in different
aspects of manufacturing from injection molding to
PCB assembly.
At the same time, we would like to learn from
existing electronics hubs so that we can share best
practices. Exchanges like the annual trip the MIT
Media Lab organizes to Shenzhen are one example of
how this might be done. Improved communication and
remote collaboration could also help bridge the cul-
tural, geographical, and language barriers. Eventually,
we imagine that additional electronics manufacturing
hubs will emerge around the world.
New Manufacturing Solutions
for Small Batches
Finally, we believe there is an opportunity to develop
new hardware manufacturing solutions optimized for
low-volume production, for batch sizes of hundreds
to thousands of units. Of course, manufacturing in
low volumes will always incur a premium due to the
14 ComputingEdge
need to amortize up-front development and tooling
costs across fewer units. Similarly, batch manufactur-
ing will never be as efficient as continuous production
because of the fixed costs associated with changing
a production line from one product to the next. How-
ever, we imagine new tools and processes that make
better use of economies of scope by amortizing nonre-
curring costs across multiple products.
Take the current difficulty of manufacturing
high-quality enclosures in low volumes as a spe-
cific example. Perhaps this could be addressed by a
modular approach that combines a standard library
of injection-molded parts in a novel way to create a
finished enclosure. Alternatively, it may be possible
for designers of new products to reuse pre-existing
injection-molding tooling, an approach that Seeed
Studio calls “design from manufacture.” Perhaps the
same reuse philosophy could be used to reduce tool-
ing costs in PCB manufacturing tests: the custom
test fixture—or jig—typically required for PCB tests
could be based on standard hardware and software
components.
Of course, the ideas above are largely speculative.
They are included simply to illustrate the potential for
innovative manufacturing solutions to address the
replication challenges.
OUTLOOK
In this paper, we have argued that the innovations
that have enabled a long tail of products in several
industries can be leveraged to do the same for hard-
ware devices. While we see these established innova-
tions as necessary, in the case of hardware, they are
insufficient. Devices bring additional complexities in
comparison with products that can be distributed dig-
itally such as apps, books, and movies. One key differ-
ence is the replication challenge, especially at low vol-
umes where economies of scale do not readily apply.
It is certainly possible to be successful with a niche
hardware product, in the same way that niche apps,
books, and movies have been possible since the incep-
tion of their respective markets. However, as things
stand, low-volume manufacturing is hard and requires
tough tradeoffs between complexity, refinement, and
price. As a result, it is all too easy to fail. We believe
that a sustained focus on the replication challenge can
reduce this complexity and risk.
July 2021

Our motivation is simple: we want to change the
dynamics in the electronic device market to enable a
diverse ecosystem of products that do not need to sell
in tens or hundreds of thousands of units in order to be
viable. Such cost-effective, low volume, batch manu-
facturing would benefit many parties across many
domains. Researchers developing custom devices
could deploy them more widely. Start-ups looking to
grow a sizeable hardware business could get to mar-
ket with less risk, giving them headroom for iteration
in search of product-market fit. Small companies could
manage a portfolio of niche, but viable, hardware
products. Large companies could be more agile and
less conservative, perhaps even trialing innovative
hardware products to learn first-hand how they fare in
the market.
We welcome feedback from the research commu-
nity on the ideas presented here and encourage others
to complement our ideas by considering how their work
might support the growth of long-tail hardware. We
will also continue to engage in dialogue with practitio-
ners in the device industry. Ultimately, a focus on long
tail hardware can overcome the tyranny of positive
feedback that currently constrains innovation, thus
creating a greater variety of viable hardware products
that address markets that are currently underserved,
and meeting the world's growing demand for interac-
tive and embedded devices.
IN MEMORIAM
This article is dedicated to the memory of Gavin Zhao
who lost his battle with cancer in August 2019. Gavin
was a manufacturing engineer at electronics man-
ufacturing company AQS, where he helped a great
many people navigate the difficulties of low volume
hardware manufacturing in China. In 2017, he was
awarded an MIT Media Lab Director's Fellowship in
recognition of his work opening the Shenzhen eco-
system up to others. Many conversations with Gavin
helped shape our thinking around long tail hardware.
ACKNOWLEDGMENTS
The authors would like to thank the many people who
have entertained our questions and ideas about elec-
tronics manufacturing over the years, and ultimately
informed the ideas presented in this paper. Partic-
ular thanks go to: Gavin Zhao, AQS, Shenzhen; Haje
www.computer.org/computingedge
SPECIAL ISSUE SPOTLIGHT
Jan Kamps, Bolt, San Francisco; Dave Vondle, Cen-
tral Standard Timing, Chicago; Bunnie Huang and
Jie Qi, Chibitronics, Singapore; Andrew Seddon, Cir-
cuit Hub, London; Jewel Deng, Coolkit, Shenzhen;
Josh Lifton and Darrell Rossman, Crowd Supply, Port-
land; Brooks Vigen, Digikey; Alex Gluhak and Ran
Katzir, Digital Catapult, London; Gus Issa, GHI Elec-
tronics, Detroit; Ji Ke and Mike Reed, Hax, Shen-
zhen; Jerry Shi, Itead, Shenzhen; Eric Klein, Lemnos
Labs, San Francisco; Liya Du, Microsoft, Shanghai;
Anita Rao and Tarun Singh, Microsoft, Redmond; Phil
Eade, Microsoft, Cambridge; Gibson Guo, Zoe He and
Nicolas Schmitt, Microsoft, Shenzhen; MJ Shen, MJ
Maker, Shenzhen; Zach Fredin, NeuroTinker, Minne-
apolis; Liam Casey and Alan Cuddihy, PCH Interna-
tional, Shenzhen; Simon Randall, Pimloc, London;
Fraser Forbes and Jonathan Smith, Premier Farnell,
Leeds; Joey Jiang, Ivy Li, Albert Miao, Eric Pan and
Shuyang Zhou, Seeed Studio, Shenzhen; Nick Bolton,
Vicon, Oxford. They would also like to thank Oliver
Amft, James Devine, Rushil Khurana, Michal Moskal,
and Nilay Patel for comments on earlier drafts of this
column and Albrecht Schmidt for his enthusiasm to
see these ideas come together.
REFERENCES
1. Wikipedia. “Zipf’s law.” Accessed: Nov. 2019. [Online].
Available: https://en.wikipedia.org/wiki/Zipf%27s_law
2. J. M. Perloff, Microeconomics, 8th ed. London, U.K.:
Pearson, 2018.
3. C. Anderson, The Long Tail: How Endless Choice is
Creating Unlimited Demand. New York, USA: Hachette
Books, 2008, ISBN 978-1401309664.
4. N. Villar, J. Scott, S. Hodges, K. Hammil, and C. Miller,
“NET Gadgeteer: A platform for custom devices,” in
Proc. Pervasive Comput., 2012, pp. 216–233.
5. S. Hodges, N. Villar, J. Scott, and A. Schmidt, “A new era
for ubicomp development,” IEEE Pervasive Comput.,
vol. 11, no. 1, pp. 5–9, Jan.-Mar. 2012. doi: 10.1109
/MPRV.2012.1.
6. J.-Y. Lo, et al., “AutoFritz: Autocomplete for prototyping
virtual breadboard circuits,” in Proc. CHI Conf. Hum.
Factors Comput. Syst., 2019, Paper 403. doi: 10.1145
/3290605.3300633.
7. F. Manjoo, The Gadget Apocalypse Is Upon Us. New
York Times, New York, NY, USA, Dec. 7 2016. [Online].
Available: https://www.nytimes.com/2016/12/07
15
SPECIAL ISSUE SPOTLIGHT
/technology/personaltech/the-gadget-apocalypse-is
-upon-us.html
8. A. Cohen, Prototype to Product: A Practical Guide
for Getting to Market. Sebastopol, CA, USA: O’Reilly
Media, Aug. 2015.
9. S. McGlaun, “CST-01 watch project may be dead”,
SlashGear, Jun. 23 2015. [Online]. Available: https:
//www.slashgear.com/cst-01-watch-project-may-be
-dead-23390137/
10. H. J. Kamps, “Hardware is Hard: Getting a Kickstarter
project out the door”, Medium Article, Jan. 22, 2015.
[Online]. Available: https://medium.com/triggertrap
-playbook/hardware-is-hard-getting-a-kickstarter
-project-shipped-59c9596bdd7f
11. H. J. Kamps, “How Triggertrap’s $500k Kickstarter
campaign crashed and burned”, Medium Article, Mar. 2,
2015. [Online]. Available: https://medium.com/@Haje
/how-a-half-million-dollar-kickstarter-project-can
-crash-and-burn-5482d7d33ee1
12. S. Hodges, et al., “SenseCam: A retrospective memory
aid,” UbiComp, 2006, pp. 177–193.
13. S. Randall, “Hacking the consumer electronics hard-
ware start-up,” Medium, Sep. 2018. [Online]. Available:
IEEE COMPUTER SOCIETY
Call for Papers
16 ComputingEdge
https://medium.com/swlh/do-not-apply-if-you-want
-an-easy-life-403e77a4a38f
14. S. Hodges, et al., “Circuit stickers: Peel-and-stick
construction of interactive electronic prototypes,” in
Proc. SIGCHI Conf. Hum. Factors Comput. Syst., 2014,
pp. 1743–1746. doi: 10.1145/2556288.2557150.
15. Andrew “bunnie” Huang, “The Hardware Hacker:
Adventures in Making and Breaking Hardware”, 2017.
16. R. Marshall, L. Archard, and S. Hodges, IoT hardware
from prototype to production, 2019. [Online]. Available:
https://aka.ms/proto-to-product
STEVE HODGES is a senior principal researcher at Micro-
soft, where he builds new embedded and interactive systems
and devices. He received a PhD in computer vision and robot-
ics from Cambridge University. Contact him at: shodges
@microsoft.com.
NICHOLAS CHEN is a senior program manager at Microsoft,
where he is building the hardware ecosystem for Azure
Sphere. He received his PhD from the University of Maryland
at College Park. Contact him at: nchen@microsoft.com.
Write for the IEEE Computer
Society’s authoritative
computing publications
and conferences.
GET PUBLISHED
www.computer.org/cfp
July 2021
PURPOSE: The IEEE Computer Society is the world’s largest
association of computing professionals and is the leading provider
of technical information in the field.
MEMBERSHIP: Members receive the monthly magazine
Computer, discounts, and opportunities to serve (all activities
are led by volunteer members). Membership is open to all IEEE
members, affiliate society members, and others interested in the
computer field.
COMPUTER SOCIETY WEBSITE: www.computer.org
OMBUDSMAN: Direct unresolved complaints to
ombudsman@computer.org.
CHAPTERS: Regular and student chapters worldwide provide the
opportunity to interact with colleagues, hear technical experts,
and serve the local professional community.
AVAILABLE INFORMATION: To check membership status, report
an address change, or obtain more information on any of the
following, email Customer Service at help@computer.org or call
+1 714 821 8380 (international) or our toll-free number,
+1 800 272 6657 (US):
• Membership applications
• Publications catalog
• Draft standards and order forms
• Technical committee list
• Technical committee application
• Chapter start-up procedures
• Student scholarship information
• Volunteer leaders/staff directory
• IEEE senior member grade application (requires 10 years
practice and significant performance in five of those 10)
PUBLICATIONS AND ACTIVITIES
Computer: The flagship publication of the IEEE Computer Society,
Computer publishes peer-reviewed technical content that covers
all aspects of computer science, computer engineering,
technology, and applications.
Periodicals: The society publishes 12 magazines and 17 journals.
Refer to membership application or request information as noted
above.
Conference Proceedings & Books: Conference Publishing
Services publishes more than 275 titles every year.
Standards Working Groups: More than 150 groups produce IEEE
standards used throughout the world.
Technical Committees: TCs provide professional interaction in
more than 30 technical areas and directly influence computer
engineering conferences and publications.
Conferences/Education: The society holds about 200 conferences
each year and sponsors many educational activities, including
computing science accreditation.
Certifications: The society offers three software developer
credentials. For more information, visit
www.computer.org/certification.
BOARD OF GOVERNORS MEETING
TBD
revised 25 May 2021
EXECUTIVE COMMITTEE
President: Forrest Shull
President-Elect: William D. Gropp
Past President: Leila De Floriani
First VP: Riccardo Mariani; Second VP: Fabrizio Lombardi
Secretary: Ramalatha Marimuthu; Treasurer: David Lomet
VP, Membership & Geographic Activities: Andre Oboler
VP, Professional & Educational Activities: Hironori Washizaki
VP, Publications: M. Brian Blake
VP, Standards Activities: Riccardo Mariani
VP, Technical & Conference Activities: Grace Lewis
2021–2022 IEEE Division VIII Director: Christina M. Schober
2020-2021 IEEE Division V Director: Thomas M. Conte
2021 IEEE Division V Director-Elect: Cecilia Metra
BOARD OF GOVERNORS
Term Expiring 2021: M. Brian Blake, Fred Douglis,
Carlos E. Jimenez-Gomez, Ramalatha Marimuthu,
Erik Jan Marinissen, Kunio Uchiyama
Term Expiring 2022: Nils Aschenbruck,
Ernesto Cuadros‐Vargas, David S. Ebert, Grace Lewis,
Hironori Washizaki, Stefano Zanero
Term Expiring 2023: Jyotika Athavale, Terry Benzel,
Takako Hashimoto, Irene Pazos Viana, Annette Reilly,
Deborah Silver
EXECUTIVE STAFF
Executive Director: Melissa A. Russell
Director, Governance & Associate Executive Director:
Anne Marie Kelly
Director, Conference Operations: Silvia Ceballos
Director, Finance & Accounting: Sunny Hwang
Director, Information Technology & Services: Sumit Kacker
Director, Marketing & Sales: Michelle Tubb
Director, Membership & Education: Eric Berkowitz
COMPUTER SOCIETY OFFICES
Washington, D.C.: 2001 L St., Ste. 700, Washington, D.C.
20036-4928; Phone: +1 202 371 0101; Fax: +1 202 728 9614;
Email: help@computer.org
Los Alamitos: 10662 Los Vaqueros Cir., Los Alamitos, CA 90720;
Phone: +1 714 821 8380; Email: help@computer.org
MEMBERSHIP & PUBLICATION ORDERS
Phone: +1 800 678 4333; Fax: +1 714 821 4641;
Email: help@computer.org
IEEE BOARD OF DIRECTORS
President: Susan K. “Kathy” Land
President-Elect: K.J. Ray Liu
Past President: Toshio Fukuda
Secretary: Kathleen A. Kramer
Treasurer: Mary Ellen Randall
Director & President, IEEE-USA: Katherine J. Duncan Director
& President, Standards Association: James Matthews Director
& VP, Educational Activities: Stephen Phillips Director & VP,
Membership & Geographic Activities:
Maike Luiken
Director & VP, Publication Services & Products: Lawrence Hall
Director & VP, Technical Activities: Roger U. Fujii
 EDITOR: David Walden, dave@walden-family.com
DEPARTMENT: ANECDOTES
Learning From Prototypes
Zbigniew Stachniak, York University
C
omputer hardware development often involves
a succession of hardware prototypes. These
prototypes are often discarded once their
functionality is tested, performance measured, and
their faults detected and analyzed. Occasionally,
functional prototypes are used for a short while
for demonstration purposes during products' prean-
nouncements or unveiling to attract the attention of
investors and technology commentators. And this is
where the life cycle of prototyping typically ends.
Fortunately, some computer prototypes survive
and end up in museums where they are preserved for
research as they may still hide the seeds of the suc-
cess or failure of both the final products and the firms
that embarked on constructing them, of technologi-
cal breakthroughs and paradigm shifts that were yet
to come.
York University Computer Museum in Toronto has
several prototypes of the MCM/70 microcomputer,
which was possibly the earliest computer mass
manufactured for personal use. The MCM/70 was
designed by a Toronto-based electronics company
Micro Computer Machines (MCM) in the early 1970s.
I have written about the MCM/70 before.1 Yet, some
key questions concerning the computer's design and
introduction to the market remained unanswered until
additional prototypes of the computer were acquired
by the museum and analyzed.
The computer was publicly demonstrated for
the first time during the APL V conference held in
Toronto on May 15–18, 1973. Before the arrival of the
prototypes at the museum, little was known about this
historic presentation. Occasional remarks about the
demo buried in oral histories gathered by the museum
describe with confidence neither the demonstrated
Digital Object Identifier 10.1109/MAHC.2020.2987408
Date of current version 29 May 2020.
18 July 2021 Published by the IEEE Computer Society 
This article originally
appeared in
vol. 42, no. 2, 2020
hardware, the scope of the demonstration, nor the
reaction of the audience to the breakthrough concept
of the portable computer for personal use. Another
question that could not be fully answered before the
resurfacing of the prototypes was how uncertainty in
the company's decision making impacted its shaping
and marketing of personal computing. In this article, I
describe how the analysis of the MCM/70 prototypes
allowed to answer these questions more fully.
EARLY MCM/70 PROTOTYPES
MCM built several prototypes of the MCM/70 before
the computer's manufacturing began in mid-1974.
All of them have their roots in the Key-Cassette con-
cept developed by the company's co-founder and
first president Mers Kutt. A drawing of it can be found
in design notes that Kutt kept from late 1971 until
mid-1974 (see Figure 1). The one-page sketch depicted
a portable computing device with built-in keyboard,
one-line display, cassette storage, and acoustic cou-
pler with built-in modem for communication over
phone lines. In addition, the Key-Cassette was to be
programmed in the APL language. The Key-Cassette
concept showed several key aspects of personal com-
puting philosophy that MCM would be developing in
the coming years—an individual-focused complete
computing environment that was easy to learn and
interact with.
The first attempt at implementing the core
Key-Cassette features was a single-board computer
put together by MCM's chief hardware engineer Jose
Laraya in mid-1972. His computer utilized an Intel
SIM8-01 simulation board which the semiconduc-
tor company offered to electronics engineers for
experimentation with its novel microprocessor and
Eprom devices. Although the Sim8-01 architecture
was inadequate to achieve MCM's design objectives,
this first prototype confirmed that building a versatile
microprocessor-based computer was feasible.2
2469-7087/21 © 2021 IEEE

FIGURE 1. Drawing of the Key-Cassette by M. Kutt (1972). Source: York University Computer Museum.
Not much is known about the next, rack-mounted
engineering prototype constructed by Laraya and his
team soon after the SIM8-01-based prototype was
declared a dead end. It was sufficiently advanced to be
demonstrated to shareholders as a proof of concept
on November 11, 1972.
At the end of 1972, the main design issue faced by
the company was an insufficiency of memory: the Intel
8008 microprocessor at the heart of the computer
could directly address only 16KB of memory while the
APL interpreter alone called for much more than that.
Furthermore, the MCM engineers had to find a way
to “compact” prototype's rack-mounted hardware to
fit it into a small enclosure planned for the portable
personal computer.
PC DEMONSTRATED
From the early stages of the MCM/70 design process,
MCM used the computer's prototypes as demonstra-
tors. In April 1972, in his corporate notes, Kutt expressed
with some urgency the need to develop a demonstra-
tor by early June 1972. Under the heading “Shortcut
to Demo,” he considered packing a power supply and
a printed circuit board (PCB) with some MCM/70 cir-
cuitry on it into a standard desktop calculator case to
have something to show to the potential investors. In
the end, MCM came up only with a cardboard mockup
which, as it turned out, sufficed to secure venture
capital from a law firm in downtown Toronto.
www.computer.org/computingedge
ANECDOTES
MCM built the first functional MCM/70 demon-
strator in early 1973, in time for the computer's public
presentation in May during the APL V conference in
Toronto. The unveiling of the MCM computer at the
conference was a landmark event in the history of
personal computing because it showed for the first
time that a practical, portable, general-purpose com-
puter designed for personal use and programmed in
a high-level language could be economically manu-
factured.3 Unfortunately, not much is known about
that demonstration. Only a brief statement about the
showing of “the first stand-alone APL microcomputer
which elicited a great deal of interest” can be found
in L.B. Moore's conference report published in APL
Quote-Quad.4 Furthermore, none of the former MCM
employees interviewed by me could describe the
demonstrated hardware or software with confidence.
That changed in 2017 when York University Computer
Museum obtained one of the MCM/70's prototypes
and a portfolio of early MCM/70 design documents.
When analyzed, these objects helped not only to iden-
tify the demonstrated prototype but also to determine,
in general terms, the presentation's content.
Among the donated documents, there are two
drawings of PCB layouts. The first of these drawings
is dated May 9, 1973, and titled “MCM 70 PROTOTYPE.”
The second drawing, dated June 26, 1973, has refer-
ence to neither a prototype name nor a revision ver-
sion. Both drawings define single-board computers,
19
ANECDOTES
FIGURE 2. Wide-case prototype of the MCM/70. MCM
promotional photograph (1973). Source: York University
Computer Museum.
i.e., computers whose main circuitry reside on a
single board. It is highly likely that the computer dem-
onstrated by MCM at the APL V conference was the
so-called “wide-case prototype” that the company
extensively used for promotional purposes in 1973
(depicted in Figure 2), and that the computer's hard-
ware was defined by the June 26th documentation.
Here is why—some early documents sent by MCM to
its shareholders stated that several MCM/70s were
constructed and ready for field trials in early May
(hence, before the APL V conference), and that by
August, MCM had built ten wide-case machines “for
marketing test purposes.” One of these documents
also includes an image which depicts exactly the
same computer as the wide-case prototype shown in
Figure 2. 5
Because of MCM's limited manufacturing capabili-
ties and the fact that the design and manufacturing of
the wide-case prototype's case took considerable
time, the production of the ten prototypes would have
had to begin before May. It is therefore reasonable to
conclude that the MCM computers available in May
were the first of the ten destined for field trials in
August. Hence, what MCM demonstrated during the
APL V conference was one of the wide-case proto-
types available in early May.
To establish the hardware makeup of the demon-
strated MCM computer, I compared the June 26th
PCB drawing with the published specifications of the
wide-case prototype and with its image (see Figure 2).
20 ComputingEdge
From the published dimensions of the wide-case proto-
type and those found on the drawing, it is evident that
the PCB depicted in the drawing would fit perfectly
into the prototype's case. Furthermore, the types,
locations, and dimensions of the keyboard, numeric
keypad, plasma display, and switches (as shown in Fig-
ure 2) match the PCB's layout exactly. Therefore, the
wide-case prototype was most likely the single-board
computer defined by the June 26th documentation. It
was built around the Intel 8008 CPU and was equipped
with 2KB of RAM and 10KB of ROM which contained,
among other software, an interpreter of a dialect of
APL—the MCM/APL.
Having the hardware identity of the demonstrated
computer established, we can turn our attention to
the question of the demonstration's content. In a 2003
interview, Gord Ramer, who implemented the MCM/
APL for the MCM/70 computer recollected that “The
demo [computer] had almost the complete APL imple-
mentation. There were still bugs to be avoided […] We
did not have a special demo version of the software,
so the demo was tightly controlled, i.e., the person on
the keyboard new what to avoid.”6 APL programming
language offers a range of built-in functions and MCM
most certainly demonstrated the capabilities of its
computer by executing programs involving some of
them. But which ones? Early MCM/70 promotional
documentation frequently included computer code
comparisons to demonstrate the efficiency of coding
in APL as opposed to programming in languages such
as Fortran and Basic. Computing the average of a set
of numbers (APL code (+/X)÷ X ← ) and sorting a set
of numbers (APL code X[ X ← ]) where given as exam-
ples. But these programs would be considered rather
elementary by an APL programmer who, instead,
would rather see examples involving APL functions
whose evaluation required considerably more time—
functions such as matrix inverse which was frequently
used for benchmarking. Unfortunately, the low speed
of the Intel 8008 processor inside the MCM/70 dem-
onstrator meant that, most likely, MCM demonstrated
programs with short execution times and only those
that could be directly entered via keyboard. Indeed,
the demonstration of long multiple-line APL programs
would require a transfer of such programs from exter-
nal storage into the computer's memory. In a prean-
nouncement document released in May 1973, MCM
July 2021
 ANECDOTES

listed an external cassette unit as an available option.

In principle, such a unit could have been interfaced
with the prototype via the computer's communication
bus called Omniport. However, it is doubtful that such
cassette storage was available for the wide-case pro-
totype at the time of its presentation in Toronto. The
computer could only be equipped with up to 2 KB of
RAM and that was not enough to support an external
cassette unit and to provide enough memory for the
storage and execution of user's applications.7 In the
end, the announced cassette unit did not materialize
as a commercial product. Instead, MCM offered three
production models of its MCM/70 computer: a model
without cassette storage and with 2 KB of RAM (as in
the wide-case prototype) and two models with built-in
cassette drives—a single cassette model with 4 KB of
RAM and a two-cassette model with 8 KB of RAM (see
Figure 4).
Scarce primary sources detailing the presenta-
tion make it difficult to ascertain the APL commu-
nity's response to the demo and to MCM's concept
of a personal APL computer. According to Moore's
above-quoted statement, the MCM/70's presentation FIGURE 3. This photograph of Ted Edwards demonstrat-
was met with interest. In a 2001 interview, Mers Kutt ing the Executive during the APL Congress in Copenhagen
made a similar comment noting that the computer appeared in Politiken on August 23, 1973. Source: York
astounded a group of IBM employees attending the University Computer Museum.
conference. In fact, two of the first dozen MCM/70
units manufactured by MCM went to IBM's General
Systems Division in Atlanta, Georgia, for the purpose evaluation of the expression in question, which was
of “research and analysis.” In 1975, IBM announced its very slow in comparison with the state-of-the-art IBM
own APL desktop computer—the IBM 5100.8 However, mainframes running APL. It is plausible that similar
other MCM employees assisting with the presenta- requests were made during the MCM/70's demonstra-
tion recollected a less enthusiastic response from tion and that the slow speed of the computer elicited
the audience who did not give the MCM/70 computer similar reactions to those recalled by Tuttle. Indeed, a
too much serious thought. Joey Tuttle, a former IBM request to evaluate an expression such as 0.7÷ 255 to
employee and a member of the IBM 5100 development generate consecutive 255 integers and to divide each
team, recalled a similar presentation of the IBM 5100 of the numbers by 0.7 could easily have resulted in a
at IBM Rochester. During the presentation, “one of the snicker as it would have taken the machine approxi-
attendees shouted out, “run +/40000 1”” which would mately 50s to evaluate it while the execution of the
create an array of length 40 000 filled with 1s and to same code on an IBM System 360 mainframe would
return the sum of all the elements stored in the array— have produced the output in a fraction of that time.
a simple way to test memory. When the presenter “Such audiences can judge harshly,” concluded Tuttle.
entered the expression and the computer seemed
frozen for an extended period of time, “the anticipa- EXECUTIVE—THE MISSING LINK
tion and the increasingly nervous body language of In the summer of 1973, several MCM/70 demonstra-
the presenter, turned to laughter.”9 In the end, the tors were on their European and North American pro-
demonstrated IBM 5100 took over 3 min to finish the motional tours. Most were the wide-case prototypes.

www.computer.org/computingedge 21
ANECDOTES
FIGURE 4. Two-cassette MCM/70 (model 708). Source: York
University Computer Museum.
But one was an altogether different piece of hard-
ware, assembled in record short time for a demon-
stration at the APL Congress that was to take place
in August at the Technical University of Denmark in
Lyngby, north of Copenhagen, Denmark. In July 1973,
MCM decided to pack the current MCM/70 hardware,
including a keyboard, a one-line plasma display (Bur-
roughs SelfScan), and a single cassette drive, into
an attaché case and to operate all of it on batter-
ies exclusively. The company expected considerable
marketing gains from the planned ground-breaking
presentation of a never-seen-before luggable, bat-
tery operated, general-purpose computer which they
named the Executive.
The gamble played off. On August 23, the day
after the Executive's demonstration, the Danish
daily Politiken published a front-page article about
a sensational computer from Canada. The article
included two photographs depicting MCM employee
Ted Edwards operating the Executive on the door-
step of the auditorium where the conference took
place (see Figure 3). “In all modesty, a real sensation
occurred yesterday when the International APL Con-
gress was about to begin at the Technical University
of Denmark.” wrote Politiken. “When the buses with
conference participants from 24 countries arrived
from hotels in Copenhagen, he [Edwards] was sitting
on the steps of an auditorium building solving com-
plex problems on his data processing machine placed
22 ComputingEdge
on his lap. Several experts who went by thought that
the computer was a joke. Many stated that such a
machine was impossible. But others who followed Ted
Edwards’ programming computations […] admitted
that the machine performed exactly as considerably
larger IBM computers running APL and connected to
an electrical outlet. There just was not any power cord
attached to Ted Edwards’ briefcase.”10
A look inside the Executive's briefcase enclosure
gives the impression that the computer's hardware
was packed in a rush using, what would be best
described as a “chain saw and duct tape” approach.11
The computer's keyboard had been crudely sawn-off
from the wide-case prototype's main board. Other
components were attached using means ranging
from nuts and bolts to sticky tape. The large ROM
board consisting of 88 Intel 1702A Eproms that barely
fit inside the case was placed at the bottom and, it
seems, was kept in place only by means of the numer-
ous wires pressing against it. The built-in cassette
drive allowed for loading and storing APL programs
and data. However crudely built, the analysis of the
Executive revealed much about the evolution of
the MCM/70 concept. The single-board approach
employed in the wide-case prototype was replaced by
a modular architecture in which computer hardware
was logically divided into interconnected individual
modules such as CPU, ROM, RAM, and interface mod-
ules that occupied separate circuit boards and com-
municated over a common bus. The Executive's CPU
and cassette interface boards were early versions of
the boards that would eventually populate the pro-
duction model of the computer. On the other hand,
the RAM board (4 KB) was, possibly, the one developed
earlier for the rack mounted prototype and would be
redesigned for the production model. Finally, the large
ROM board inside the Executive was the same as the
one used in the early production models. Because in
early 1974, ROMs containing MCM/70's systems soft-
ware were not yet delivered by the supplier (Electronic
Arrays), the first MCM computers were equipped with
the same ROM board found in the Executive and,
because of its large size, it was mounted externally
under the bottom of the computer's case (thus earn-
ing it a nick-name “pregnant bottom”). This shows
that from the hardware architecture point of view,
the Executive was almost identical to the production
July 2021
 ANECDOTES

model and, hence, that by July 1973, the company had

already abandoned the single-board approach repre-
sented by the wide-case prototype.
For a brief time, MCM contemplated manufactur-
ing the Executive and made announcements to that
effect in the press. In October 1973, the computer was
shown during the National Computer Show and Con-
ference in Toronto but, in the end, the company quietly
abandoned the Executive concept. The surviving MCM
corporate documents do not provide any justification
for the decision. However, some arguments against
the Executive can be reconstructed by analyzing its
design and another prototype put together by Edwards
some years later. To begin with, the Executive was an
attempt at building a fully luggable, battery-operated,
personal data processing system. It was to offer com-
plete data processing and communication functional-
ity at the user's fingertips. However, most if not all of
the Executive's features were planned for the MCM/70
desktop which, in addition, was not much larger or FIGURE 5. Ted Edwards’ Executive-E. Source: York University
heavier than the Executive. Like the Executive, it could Computer Museum.
be operated on batteries and carried around in an
elegant leather case that could be purchased for $150
from MCM. left for anything at all, not even an internal fan, which
Most likely, there were also arguments of a tech- brings us to the second issue—heat dissipation. The
nical nature against the Executive. Some of these Executive's case had no provisions to dissipate heat
arguments can be deduced from the Executive-like during the computer's operations. In the MCM/70 pro-
computer designed and built by Edwards around duction model, which also lacked an internal fan, the
1975–1976 (see Figure 5; I will refer to this computer heat dissipation problem was solved by packaging all
as the Executive-E.).12 From a hardware point of view, the boards in special aluminum casings sandwiched
the Executive-E is an MCM/70 clone. All the PCB together to form the back end of the computer and to
boards used in it are the same as those found in the act as a giant heat sink. In the Executive-E, Edwards
production model of the MCM/70 or its MCM/700 used the entire aluminum case of the computer as a
refinement. However, from a packaging perspective, hit sink.
it is the Executive repackaged into an all-aluminum It is likely that the Executive-E was an attempt by
case. The addition of the second cassette drive may Edwards at salvaging the Executive concept. Because
suggest that one of the issues with the Executive was of its sturdy packaging it could have been marketed
its limited memory and, as a consequence, limited as a luggable data collection and processing system
functionality. While the Executive used its single cas- for use in harsh rugged environments. However, by
sette drive for external storage, the amount of RAM the time its design was finished, Edwards was no
that could be made available to a user remained low. longer with MCM. It would be left to another com-
MCM solved the insufficient RAM problem by develop- pany to build and successfully introduce a portable
ing a cassette-based virtual memory system (called battery-operated computer for rugged environments.
AVS) that made over 100 KB of memory available to GRiD Systems, Inc., introduced its first laptop–the
applications. But AVS required a dedicated cassette GRiD Compas 1001–in 1981. In the following years,
drive and there was simply no space left in the Execu- several GRiD laptops would find their way into space
tive's case for the second drive. There was no space supporting NASA's early Shuttle missions.13

www.computer.org/computingedge 23
ANECDOTES
MARKETING
PERSONAL COMPUTING
With the MCM/70 personal computer concept, MCM
hoped to create a lucrative niche on the electronics
market between the inexpensive, easy-to-use desktop
electronic calculators that were inadequate for many
data processing tasks, and large, expensive, and com-
plex to operate general-purpose computers. To suc-
ceed, the company needed to support its personal
computing venture with a well-defined and appealing
product philosophy and attractive marketing to pro-
mote it. MCM promotional documents from the mid–
1973 paint an interesting picture of how the company
was stitching together a personal computing para-
digm from the computer's basic features as well as its
hardware and software options. An August 24, 1973
portfolio of MCM/70 preannouncement documents
distributed to MCM shareholders already contained
an outline of the paradigm.
The revolutionary concept of the MCM/70 is that
it brings to the world of computing what the $100
hand held calculator brought to the world of
calculating. [It is] of a size, price, and ease-of-use
as to bring personal computer ownership to
business, education, and scientific users previ-
ously un-served by the computer industry.14
But that's not all. The portfolio also listed the
MCM/70's basic features together with available
options. The list was a very long one: two types of
printers (one external and one built into the comput-
er's chassis), cassette drives, serial communications
interface, CRT (Cathode Ray Tube) display, RAM expan-
sion of up to 64 KB, and even plug-in ROM modules
with preprogrammed applications software. These
features and options were designed to capture the
core features of the personal computing environment
that MCM wanted to offer with its desktop. However,
throughout 1973 and 1974, the list of the MCM/70's
available options was a moving target, continuously
revised with some options added, others dropped
to reappear again. Were these changes reflective of
uncertainty in company's decision-making or of an
evolving point of view on personal computing that
necessitated the shifting of priorities for the devel-
opment of at least some of these options? Or were
24 ComputingEdge
they part of a marketing ploy to overwhelm potential
customers and MCM shareholders with claims of its
offering's sophistication and completeness, an early
example of vaporware? The analysis of the surviving
MCM hardware and corporate documents shows that
all of these concerns were the case.
In late 1973, MCM dropped the preprogrammed
ROMs option all together, gave up on memory exten-
sion to 64KB, replaced the built-in printer option
(which had never materialized) with an external impact
printer. But not all of the company's claims were a
marketing gambit. The Executive's hardware already
featured a partially implemented communications bus
(Omniport) that allowed the computer, in principle, to
interface with printers.15 However, what the company
did not have at that time was the printer itself, and
it was not until mid-1975 when, finally, it offered its
MCP-132 printer (the Diablo HyType I printer). At that
time, MCM also upgraded the MCM/70 with an EAI
communications interface that allowed the computer
to communicate with a range of other peripherals.
MCM's stated objective “to offer a complete micro
computer system” was finally realized.16
The MCM/70's built-in display (Burroughs Self-
Scan) was one of the core features of the all-in-one
hardware concept. While such a display enhanced
portability, the single-line 32-character-long solu-
tion adopted for the computer offered a convenient
display environment for only rudimentary APL calcu-
lations. MCM maintained that MCM/70's display was
sufficient for many tasks because of APL's coding
efficiency. For all other applications, there was to
be a CRT display, already listed in August 1973 as an
available option. Of course, the company could have
equipped the MCM/70 with a multiple-line display, as
was done for some calculators of the era, if it were not
for the fact that the computer required a dedicated
segment of RAM for storing characters to be displayed
(display memory). More display lines would take away
precious RAM from the user's work space. The RAM
shortage was so severe that display memory was also
used for temporary storage during computations. A
first-time user would have been quite surprised and
astounded when after entering a command, such as
the above mentioned 0.7 ÷ 255, the computer would
begin a 50-second-long display of strange and dazzling
patterns (i.e., visual representations of temporal data
July 2021

stored in display memory) before eventually clearing
everything and showing the result of the computa-
tion. Therefore, having a single line display built-in and
an external CRT terminal as an option seemed like a
reasonable solution. But the CRT option disappeared
from MCM's 1974 promotional literature and was not
discussed (as a work in progress) in any surviving min-
utes from MCM managers’ meetings. It would not be
until 1976 that MCM finally introduced such a display
for its new computer—the MCM/800. This might sug-
gest that the announcement of a CRT display as an
“available” option made three years earlier was, to put
it mildly, a careless instance of the company “snowing”
customers and shareholders with options that were
only meant to enhance the MCM/70's image. Or is it?
To answer this question, I analyzed the ROM boards
inside the MCM/70, Executive-E, and MCM/800
computers. The inspection of these boards showed
that the same three ROM sockets (out of 16) were left
unpopulated in the MCM/70's and the Executive-E
while similar boards inside the MCM/800s had all
16 ROM chips installed. It turns out that these three
empty sockets were reserved for correcting and future
expansion of the MCM/70's systems software, includ-
ing the addition of CRT support. One may therefore
conclude that financial or other corporate difficulties
that MCM experienced in 1974 forced the company to
drop several options, including an inexpensive printer
and a CRT display, in order to concentrate on the
MCM/70's prompt introduction to the market. How-
ever, as evident from the MCM/70's ROM board, some
necessary hardware provisions for such options were
made at the start.
LOOK OF A GADGET
The MCM/70 concept was shaped by ideas coming
from several sources, one of which was the calcula-
tor industry. From the late 1960s, desktop programma-
ble calculators were presented as minicomputers and
even as minis for personal use, as was the case with
the Olivetti Programma 101 introduced in 1965. Fur-
thermore, the manufacturers of hand-held calcula-
tors marketed their devices as consumer electronics
gizmos. The MCM/70 was to be both; a stylish “gizmo”
look was regarded just as important a feature of the
MCM/70 as its conception for personal use. Through
the industrial design of the MCM/70 the company
www.computer.org/computingedge
ANECDOTES
wanted to draw attention to a new computing para-
digm and to create its own unique identity.
In his 1965 article “The Great Gizmo,” Reyner Ban-
ham characterized a gizmo as
[…] a small self-contained unit of high performance
in relation to its size and cost, whose function
is to transform some undifferentiated set of
circumstances to a condition nearer human
desires. The minimum of skills is required in its
installation and use, and it is independent of any
physical or social infrastructure beyond that
by which it may be ordered from a catalogue
and delivered to its prospective user.17
As early as the MCM/70's prototyping stage, the
company's description of the computer provided
an almost undeviating instance of Banham's char-
acterization of a gizmo. The Key-Cassette concept
was a gizmo (although only on paper), and so was the
Executive. The design of the wide-case prototype
accomplished not only the requirement of hosting
all the necessary hardware in a single box but also of
finding the right balance between stylish eye-catching
design, functionality, and practicality. Two years of
design experiments culminated in the 1974-release
of the MCM/70's production model which inherited
the all-in-one concept from the Key-Cassette and
the Executive, and a defining stylish design from the
wide-case prototype. In the late 1970s, the nascent
home computer industry would follow in the footsteps
of MCM choosing appealing, dashing designs over the
industrial look of minicomputers.
REFERENCES
1. See Z. Stachniak, Inventing the PC: The MCM/70 Story,
McGill-Queen’s University Press, 2011, and Z. Stach-
niak, MCM on Personal Software, IEEE Ann. History
Comput., vol. 39, no. 1, pp. 29–51, 2017.
2. For more information on this prototype, see Z. Stach-
niak, “SIM8-01: A proto PC,” IEEE Ann. History Comput.,
vol. 29, no. 1, pp. 34–48, 2007.
3. Three months earlier, a French company Réalisa-
tions Études Électroniques (R2E) announced its
microprocessor-based Micral (see, Bui, R. Un minior-
dinateur pour moins de 8500FF, zero.un.informatique
hebdo, no. 228, Feb. 12, 1973, pp. 1 and 5.). However, the
25
ANECDOTES
Micral was not intended to be a personal computer:
“MICRAL’s principal use is in process control. It does
not aim to be an universal mini-computer,” Micral
User’s Manual, R2E, Jan. 1974, p. 66.
4. L.B. Moore, A report of APL V Conference, APL
Quote-Quad, Jun. 1973, pp. 20–21.
5. The documents in question are dated May 4 and Aug.
24, 1973. York University Computer Museum, MCM
Collection.
6. Author interview with Gord Ramer, Jan. 15, 2003.
7. A large portion of the computer’s 2 KB of RAM was
used for display, APL execution stack, and a variety of
tables required by the computer’s operating system.
Interfacing a cassette drive with a computer would
require an allocation of additional RAM to store
information about the interfaced device and the tape’s
content leaving little space for anything else.
8. I have encountered no evidence for the design of the
IBM 5100 being, in any way, influenced by the MCM/70.
9. J. Tuttle, private communication, Jan. 2020.
10. Politiken, Aug. 23, 1973, pp. 1 and 20.
ADVERTISER INFORMATION
Advertising Coordinator
Debbie Sims
Email: dsims@computer.org
Phone: +1 714-816-2138 | Fax: +1 714-821-4010
Advertising Sales Contacts
Mid-Atlantic US:
Dawn Scoda
Email: dscoda@computer.org
Phone: +1 732-772-0160
Cell: +1 732-685-6068 | Fax: +1 732-772-0164
Southwest US, California:
Mike Hughes
Email: mikehughes@computer.org
Cell: +1 805-208-5882
Northeast, Europe, the Middle East and Africa:
David Schissler
Email: d.schissler@computer.org
Phone: +1 508-394-4026
26 ComputingEdge
11. The Executive was donated to York University
Computer Museum in 2017.
12. The date codes found on its ICs, suggest that the
computer was put together around 1976. Because
Edwards left MCM at the end of 1975, he continued the
design after leaving MCM.
13. Accessed on: Mar. 2020. [Online]. Available: https:
//airandspace.si.edu/node/35305
14. MCM/70 pre-announcement shareholder documents,
Aug. 24, 1973. York University Computer Museum,
MCM collection.
15. This was confirmed using the MCM/70 emulator
developed at York University Computer Museum and
which uses almost identical systems software to that
found in the Executive’s ROMs.
16. MCM Newsletter, no. 1, 1976, pp. 2 and 3. The MCP-132
printer plotter was sold by MCM for $4500 which was
almost the same price as the MCM/70 in its basic
configuration ($4970).
17. R. Banham, The Great Gizmo, Ind. Des., vol. 12, pp.
48–59, 1965.
Central US, Northwest US, Southeast US, Asia/Pacific:
Eric Kincaid
Email: e.kincaid@computer.org
Phone: +1 214-553-8513 | Fax: +1 888-886-8599
Cell: +1 214-673-3742
Midwest US:
Dave Jones
Email: djones@computer.org
Phone: +1 708-442-5633 Fax: +1 888-886-8599
Cell: +1 708-624-9901
Jobs Board (West Coast and Asia), Classified Line Ads
Heather Bounadies
Email: hbuonadies@computer.org
Phone: +1 623-233-6575
Jobs Board (East Coast and Europe), SE Radio Podcast
Marie Thompson
Email: marie.thompson@computer.org
Phone: +1 714-813-5094
July 2021
IEEE
Computer
Society Has
You Covered!
WORLD-CLASS CONFERENCES — Stay
ahead of the curve by attending one of our
215+ globally recognized conferences.

DIGITAL LIBRARY — Easily access over 800k

articles covering world-class peer-reviewed
content in the IEEE Computer Society
Digital Library.

CALLS FOR PAPERS — Discover

opportunities to write and present your
ground-breaking accomplishments.

EDUCATION — Strengthen your resume

with the IEEE Computer Society Course
Catalog and its range of offerings.

ADVANCE YOUR CAREER — Search the

new positions posted in the IEEE Computer
Society Jobs Board.

NETWORK — Make connections that count

by participating in local Region, Section,
and Chapter activities.

Explore all of the member benefits

at www.computer.org today!
 DEPARTMENT: EXPERT OPINION This article originally
appeared in

Monolithically Integrated RRAM-

and CMOS-Based In-Memory
vol. 39, no. 6, 2019

Computing Optimizations for

Efficient Deep Learning
Shihui Yin, Arizona State University
Yulhwa Kim, Pohang University of Science and Technology
Xu Han and Hugh Barnaby, Arizona State University
Shimeng Yu and Yandong Luo, Georgia Institute of Technology
Wangxin He, Arizona State University
Xiaoyu Sun, Georgia Institute of Technology
Jae-Joon Kim, Pohang University of Science and Technology
Jae-sun Seo, Arizona State University

Resistive RAM (RRAM) has been presented as a promising memory technology toward deep
neural network (DNN) hardware design, with nonvolatility, high density, high ON/OFF ratio,
and compatibility with logic process. However, prior RRAM works for DNNs have shown
limitations on parallelism for in-memory computing, array efficiency with large peripheral
circuits, multilevel analog operation, and demonstration of monolithic integration. In this
article, we propose circuit-/device-level optimizations to improve the energy and density of
RRAM-based in-memory computing architectures. We report experimental results based
on prototype chip design of 128 × 64 RRAM arrays and CMOS peripheral circuits, where
RRAM devices are monolithically integrated in a commercial 90-nm CMOS technology. We
demonstrate the CMOS peripheral circuit optimization using input-splitting scheme and
investigate the implication of higher low resistance state on energy efficiency and robustness.
Employing the proposed techniques, we demonstrate RRAM-based in-memory computing
with up to 116.0 TOPS/W energy efficiency and 84.2% CIFAR-10 accuracy. Furthermore, we
investigate four-level programming with single RRAM device, and report the system-level
performance and DNN accuracy results using circuit-level benchmark simulator NeuroSim.

D
eep learning algorithms have shown tre-
mendous success in recent years1 for vari-
ous applications including computer vision,
speech recognition, language translation, etc.
However, an increasing gap exists between the
exponential network size growth of state-of-the-art
DNNs (e.g., tens of millions of parameters) and the
incremental energy-efficiency improvement of
Digital Object Identifier 10.1109/MM.2019.2943047
Date of current version 8 November 2019.
conventional memory technologies (e.g., CMOS scal-
ing) for hardware accelerator designs.2
To bridge this gap and largely improve the
memory energy efficiency, in-memory computing
(IMC) has been proposed in recent years across
different memory technologies. 3–9 IMC typically
asserts multiple or all rows simultaneously to perform
multiply-and-accumulate (MAC) computations of
DNNs inside the memory, e.g., along the bitlines with
analog current/voltage.
SRAM-based IMC works3–5 demonstrate high-
energy efficiency, however typically such IMC SRAM

28 July 2021 Published by the IEEE Computer Society  2469-7087/21 © 2021 IEEE

bitcells include a few additional transistors, which
degrades density and leakage. In addition, custom
peripheral circuits such as analog-to-digital convert-
ers (ADCs) incur lower array efficiency. Since one
SRAM cell occupies 150–300 F2 (F is the feature size
of a technology node), on-chip SRAMs cannot hold all
weights of DNNs. Therefore, CMOS hardware acceler-
ators inevitably involve off-chip DRAMs at the system
level, which results in high energy consumption.
Consequently, a number of works have proposed to
bring computation closer to the DRAM. DRAM-based
near-memory computing proposes to add logic in the
DRAM die, however logic capability in the optimized
DRAM process is relatively limited. On the other hand,
DRAM-based IMC is more challenging, because the
conventional 1T1C DRAM read is destructive, and thus
requires additional overheads such as data copy and
write back.6 DRAM cell designs with nondestructive
read have been proposed (e.g., 2T1C, 3T1C),7 but they
directly degrade density, which is especially disadvan-
tageous for area-efficient DRAMs.
In addition, both SRAM and DRAM are volatile
and have increasing concerns on leakage power in
scaled CMOS nodes. To that end, resistive nonvolatile
memory (NVM) has emerged as a good alternative
due to high density, nonvolatility, and nondestructive
read. Among several well-known candidates including
phase change memory (PCM), resistive RAM (RRAM),
and magnetic RAM (MRAM), this article focuses on
RRAM owing to its high ON/OFF ratio, multilevel pro-
grammability, and monolithic integration capability.
There has been only a few works that have demon-
strated monolithically integrated RRAM and CMOS for
DNN hardware design.8–10
Mochida et al.8 designed 180-nm and 40-nm pro-
totype chips with embedded RRAM arrays. However,
only simple multilayer percepton (MLP) has been dem-
onstrated that resulted in low-inference accuracy of
90.8% for MNIST data set. An RRAM macrointegrated
with multilevel sense amplifiers (SAs) in 55-nm CMOS
logic process was recently reported by Xue et al.,9
targeting convolutional neural networks (CNNs). How-
ever, a relatively low CNN accuracy of 81.83% accuracy
for CIFAR-10 data set was achieved with binary/ter-
nary precision. Moreover, only nine WLs are asserted
simultaneously in the 256 × 512 subarray, which limits
further parallelism, and a relatively complex 4-bit ADC
www.computer.org/computingedge
EXPERT OPINION
was employed at the RRAM array periphery, degrad-
ing array efficiency and energy consumption. In the
article by Shulaker et al.,10 a monolithically integrated
3-D nanosystem has been presented, which connects
CMOS transistors, carbon nanotube transistors
(CNFET), and RRAM devices in different layers with
inter-layer vias. A small-scale support vector machine
accelerator has been demonstrated, but applicabil-
ity for larger DNNs has not been shown. While there
has been considerable improvement in the CNFET
BOTH SRAM AND DRAM ARE VOLATILE
AND HAVE INCREASING CONCERNS
ON LEAKAGE POWER IN SCALED
CMOS NODES. TO THAT END, RESISTIVE
NONVOLATILE MEMORY (NVM) HAS
EMERGED AS A GOOD ALTERNATIVE
DUE TO HIGH DENSITY, NONVOLATILITY,
AND NONDESTRUCTIVE READ. AMONG
SEVERAL WELL-KNOWN CANDIDATES
INCLUDING PHASE CHANGE MEMORY
(PCM), RESISTIVE RAM (RRAM), AND
MAG- NETIC RAM (MRAM), THIS ARTICLE
FOCUSES ON RRAM OWING TO ITS
HIGH ON/OFF RATIO, MULTILEVEL
PROGRAMMABILITY, AND MONOLITHIC
INTEGRATION CAPABILITY.
integration with CMOS or RRAM, in terms of manufac-
turability and yield, integration of RRAM with CMOS in
commercial technology is much superior.11
In this article, we address such limitations in
RRAM-based IMC toward energy-/area-efficient and
accurate DNN hardware design, using monolithic
integration of RRAM and CMOS. In particular, we
investigate three different device/circuit techniques:
1) modulating resistance values for binary RRAM
devices; 2) peripheral circuit minimization with
input-splitting technique; and 3) multilevel RRAM pro-
gramming. We report measurement results of 90-nm
CMOS prototype chip in monolithically integrated
RRAM arrays, which executes IMC operations of
CNNs for CIFAR-10 data set.
In our IMC architecture, monolithic integration
of RRAM and CMOS is crucial, since we need dense
29
EXPERT OPINION
FIGURE 1. Prototype chip design with monolithically integrated RRAM and 90-nm CMOS technology14 (adapted with permis-
sion). This work presents further energy/area optimization.
connections to all wordlines (WLs) and bitlines of
the RRAM array. If RRAM and CMOS are not mono-
lithically integrated (e.g., using through-silicon vias or
silicon interposers), the bitline and WL delays will be
excessive and the integration density will be too low.
Furthermore, monolithic integration of RRAM with
CMOS is simpler and less expensive than that with
CNT.10 RRAM process is CMOS fabrication compat-
ible, with just a few layers of oxide deposition at the
contact via at back-end-of-line (BEOL) compatible
temperature. Typically, only one additional mask/
lithography is required, allowing RRAM integration to
be low cost.
RRAM PROTOTYPE CHIP DESIGN
We designed a prototype chip for RRAM-based robust
IMC with Winbond's embedded RRAM technology,11
which monolithically integrates 90-nm CMOS and
RRAM between M1 and M2 [see Figure 1(a)]. Figure 1(b)
shows the pad-limited chip micrograph and the core
area of the chip. As shown in the top-level block dia-
gram in Figure 1(c), the chip design includes a 128 ×
64 1T1R array, row decoder, level shifter, eight 8-to-1
column multiplexers, eight 3-bit flash ADCs based
on seven voltage-mode SAs, and two 64-to-1 column
decoders for RRAM cell-level programming. The row
30 ComputingEdge
decoder has two modes of operation: 1) turning on all
WL signals simultaneously for binary or low-precision
MAC operation; or 2) generating one-hot WL signals
for cell-level programming. As shown in Figure 1(d),
ADCs and column multiplexers consume a large por-
tion of the core area. In this article, further energy/
area optimization is investigated including periph-
eral circuit minimization by using higher LRS and
input-splitting scheme.
Conventional binary RRAMs cannot effectively rep-
resent the positive and negative weight values (+1 and
–1) in binarized neural networks (BNNs),12 because the
high-resistance state (HRS) and low-resistance state
(LRS) values of binary RRAM devices are both posi-
tive. In addition, as shown in Figure 2, the activation/
weight value combinations of +1/+1 and –1/–1 should
result in the same effective resistance. To that end, we
proposed to use a “XNOR-RRAM” bitcell design13,14 for
BNNs. As shown in Figure 2, the XNOR-RRAM bitcell
is designed with differential RRAM cells and differ-
ential WLs. The binary activations are implemented
with the differential WLs, and the binary weights are
implemented with the HRS/LRS values of XNOR-RRAM
bitcells. With all differential WLs asserted simultane-
ously, all cells in the same column in parallel compute
the binary MAC operations. Since one XNOR-RRAM
July 2021

FIGURE 2. IMC operation of XNOR-RRAM14 (adapted with permission).
cell consists of two 1T1R bitcells, 128 × 64 1T1R array
effectively represents 64 × 64 XNOR-RRAM cells.
Both the preliminary simulation results13 and ini-
tial measurement results14 of the XNOR-RRAM design
only considered the default LRS and HRS values for the
binary RRAM devices, and employed a 3-bit ADC at the
periphery for digitizing the analog partial MAC value. In
this article, we investigate three further optimizations
in monolithically integrated RRAM devices and periph-
eral circuits, toward enhancing the energy efficiency
and density of the RRAM-based IMC systems.
First, since the default LRS value (~6 k ) consumes
large current and the ON/OFF ratio is relatively high
(~150), we explore using higher LRS values (e.g., ~12
and ~24 k ) to evaluate the tradeoff between current
reduction, ON/OFF ratio, and CNN accuracy.
Second, although a 3-bit ADC is relatively simple,
it still consumes a large area compared to the RRAM
array itself, resulting in low-array efficiency. We
present further algorithm/hardware improvements
beyond the previous input-splitting techniques,15 and
employ binary SAs with an unified reference voltage
across all columns, instead of ADCs at the RRAM array
periphery, for digitizing the analog partial MAC values.
www.computer.org/computingedge
EXPERT OPINION
Considering that tightly spaced reference voltages
make flash ADCs more susceptible to variability at low
voltages, we show that the proposed input-splitting
scheme actually results in much improved accuracy at
lower supplies.
Finally, beyond binary RRAM devices, we inves-
tigate four-level programming with the same RRAM
devices in our prototype chip, and experimentally
validate the density, energy, and performance gains by
benchmarking a CNN for CIFAR-10 data set.
HIGHER RESISTANCE FOR
LRS DEVICES
In binary RRAM devices, only two states per device
exist, namely LRS (high conductance) and HRS (low
conductance). In commercial RRAM technologies that
are typically used for storage applications, ON/OFF
ratio of higher than 100 has been reported. Having a
large ON/OFF ratio is certainly good, but on the other
hand, having high conductance value for the LRS leads
to high current consumption.
To that end, for a given HRS value fixed, and if we
have higher LRS values in binary RRAM devices, then
the current and energy consumption could be largely
31
EXPERT OPINION
FIGURE 3. New input-splitting scheme that allows unified reference voltage for all SAs in the RRAM array periphery.
reduced. On the other hand, compared to the default
LRS, targeting LRS to have a higher resistance value
can result in wider distribution after programming
or more susceptible to nonideal effects such as read
disturb. In addition, and if the LRS and HRS ranges
become relatively close, it will adversely affect the
DNN accuracy for the RRAM-based IMC hardware.
PERIPHERAL CIRCUIT
MINIMIZATION WITH INPUT-
SPLITTING SCHEME
Input splitting is a method of the BNN architecture
design for ADC-free IMC.15 Input splitting recon-
structs a large BNN layer with a network of small lay-
ers. It splits input of a large layer so that the number of
inputs per split group is less than or equal to row count
of the given RRAM array. Each split group constructs a
new small layer, and the binary output generated from
small layers is accumulated and subsequently bina-
rized with a threshold value of zero. Then, each layer of
input-split BNN can fit on RRAM array so that the array
can generate binary neuron values as output values.
However, batch normalization governs that each neu-
ron has its own threshold value, which necessitates
each column to have a digital-to-analog converter,4
adding a large overhead.
In this article, we modified the conventional
input-splitting method15 to eliminate columnwise
32 ComputingEdge
threshold values. Batch normalization conducts scal-
ing and shifting operation, and the shifting operation
generates threshold values. Therefore, as illustrated
in Figure 3, we removed batch normalization before
output binarization of small layers. Instead, we experi-
mentally found a proper scaling factor for prebinariza-
tion values of small layers. For the RRAM array with 64
rows, we found that, by scaling prebinarization value
with 1/20, most of scaled values lie in the range of [–1,
1]. As there is no shifting operation on prebinarization
value of small layers, the columnwise threshold is fixed
to 0. Then, we added batch normalization after the
merge to compensate for the loss of batch normaliza-
tion on small layers.
We tested a VGG-like CNN for CIFAR-10 data-
set, which has the network structure of input-
128C3-128C3-MP2-256C3-256C3-MP2-512C3-512C3-
MP2-1024FC-1024FC-10FC.12 Here, 128C3-128C3 refers
to the convolution layer with 128 input feature maps, 3
× 3 kernels, and 128 output feature maps, MP2 refers to
2 × 2 max-pooling, and 1024FC refers to the fully con-
nected layer with 1024 hidden neurons.
As we used RRAM arrays with 64 effective rows,
the input counts per input-split BNN layer was set to
63 for convolution layers and 64 for fully connected
layers. We used 63 for convolution layer because we
use 3 × 3 kernel for convolution, and 63 is the clos-
est value less than equal to 64. In addition, to make
July 2021

FIGURE 4. Conductance distribution is shown for four levels of RRAM device programming. Both measurement data from proto-
type chip and fitted Gaussian distribution curves are shown.
the input of convolution layer be divided by 63, we
changed the number of channels to be an integer mul-
tiple of 7. Using Torch, we trained the input-split BNN
with the same training condition used in conventional
input splitting.15 For comparison, we trained baseline
BNN (nonsplit BNN), input-split BNN with columnwise
threshold, and input-split BNN without columnwise
threshold. The algorithm simulation results showed
that the input-split BNN without columnwise thresh-
old model has compatible accuracy (86.64%) with
the baseline BNN (88.46%) and input-split BNN with
columnwise threshold (88.24%).
MULTILEVEL RRAM DEVICES
Multilevel Programming Scheme
To achieve 2-bit RRAM, two more conductance states
are inserted between minimum and maximum con-
ductance levels so that the conductance interval is
equal between adjacent states. A write–verify pro-
gramming scheme is iterated until less than 2% of
RRAM cells are outside the target conductance range
for each of the four levels. The maximum number of
write–verify iterations to program one RRAM cell is
specified as Nmax . For each conductance state, 4096
RRAM cells in the prototype chip are programmed
and measured. It is observed that the conductance
distribution becomes more concentrated as Nmax
increases. The Nmax to achieve the target conduc-
tance range are 15, 30, 15, and 10 for the four conduc-
tance states, respectively. After programming, the
www.computer.org/computingedge
EXPERT OPINION
percentage of the RRAM cells that are outside the
target conductance ranges were 0.32%, 1.32%, 0.92%,
and 0.44%, respectively.
Inference Accuracy Simulation
The inference accuracy for a CNN is simulated with the
measured 2-bit RRAM data. However, considering the
limited measurement data (4096 data points for each
state) compared to the total number of parameters
in a CNN, we first fitted the probability density func-
tion (PDF) of the measured conductance data with a
linear combination of multiple Gaussians as the fitted
PDF. Then, the conductance values were generated
with the fitted PDF for a large CNN. Figure 4 shows the
PDF of the measured conductance and the conduc-
tance values generated with fitted PDF. The distribu-
tion tails of the experiment data are captured with the
fitted PDF.
Using 2-bit weights and 4-bit activations, we
benchmarked the same VGG-like CNN for CIFAR-10.
It is assumed that each 2-bit weight is stored into one
RRAM cell. We first trained the CNN with the quan-
tized training method proposed by Wu et al.,16 and
obtained the software baseline accuracy of 91.7%. The
2-bit weights are then mapped to conductance states,
where the conductance values of each RRAM cell are
generated with the fitted PDFs of the corresponding
states. The inference accuracy is simulated for three
different array size 64 × 64, 128 × 128, and 256 × 256,
where we employed flash ADCs with 5-bit precision
using nonlinear quantization.13
33
EXPERT OPINION
FIGURE 5. Measured ADC output results compared with bitcount values from BNN algorithm.
MEASUREMENT AND
SIMULATION RESULTS
Binary-RRAM-Based IMC Energy
and Accuracy Characterization with
Higher LRS and Input Splitting
We envision that large binary CNNs are mapped onto
multiple RRAM arrays, where weights for different
input channels are stored on different rows, weights
for different output channels are stored on different
columns, and weights within each convolution kernel
(e.g., 9=3 × 3) are stored in different RRAM macros. 3, 14
Subsequently, the partial MAC results from different
RRAM macros are accumulated via digital simulation.
In the article by Yin et al.,14 3-bit ADC was used to dig-
itize the analog partial MAC values, where seven refer-
ence voltages for each flash ADC required offset can-
cellation in order to achieve >83% CIFAR-10 accuracy.
The new input-splitting scheme presented in this arti-
cle substantially reduces such calibration overhead,
since we only need binary SAs to digitize the analog
partial sum, and the same reference voltages are used
for all 64 columns of the RRAM array.
Another important challenge for the flash ADC is
that the adjacent reference voltages are very close to
each other, especially since the partial sum data distri-
bution is concentrated near zero.13 If we lower the sup-
ply voltage, the reference voltages actually become
even closer to each other, which makes it more
34 ComputingEdge
susceptible to process variation. On the other hand,
since the input-splitting scheme allows us to have only
one reference voltage for the SAs, the digitization is
inherently more robust to variability and noise.
We performed chip measurements for the experi-
ments of higher LRS values and the input-splitting
scheme. For the higher LRS experiment, we pro-
grammed RRAM devices with different target LRS val-
ues of 6, 12, and 24 k . For the input-splitting scheme
experiment, with the same XNOR-RRAM prototype
chip, we only use one SA out of the seven SAs that are
present in the flash ADC. This means that when we
employ the input-splitting scheme, the overall ADC
area in the RRAM macro is effectively reduced by 7×.
In Figure 5, we show the comparison of the bitcount
values from the BNN algorithm (i.e., ideal partial sum
values) and the measured ADC output values using
12-k\ LRS target, for both the conventional scheme
with 3-bit ADCs and the input-splitting scheme with
binary SAs. As we compare the 1.2- and 0.8-V supply
results, it can be seen that the ADC output values
become less accurate at 0.8 V. However, with a single
reference level, the input-splitting scheme still main-
tains more robust operation even at lower voltages.
As shown in Figure 6, the energy efficiency
(TOPS/W) of RRAM-based IMC increases with higher
LRS values and with lower supply voltages. The
CIFAR-10 accuracy values for the VGG-like CNN with
voltage scaling are reported in Figure 7 with different
July 2021
 EXPERT OPINION

LRS values for both the input-splitting scheme with

binary SAs and the conventional scheme with 3-bit
ADCs. For the conventional scheme with ADCs, it
can be seen that the CIFAR-10 accuracy degrades by
a large amount when the supply voltage scales below
the nominal 1.2 V. This is due to the fact that the seven
reference voltages for the flash ADC are separated
only by a small voltage value, which aggravates with
lower supply voltages, incurs more ADC output errors,
and adversely affects the DNN accuracy.
For the input-splitting scheme, there is only one
reference voltage used by all eight SAs for 64 columns;
the SA operation is more robust against voltage FIGURE 6. Energy efficiency with voltage scaling for RRAM
scaling, noise, and variability. As a result, Figure 7 IMC with different LRS values.
shows that high CNN accuracy is
maintained for the input-splitting
scheme for 12-k /24-k LRS val-
ues, down to 0.8-V supply. The
input-splitting scheme also shows
higher accuracy for cases when
RBL voltage is around 0.6–0.7 V
(high gain region for SA with dif-
ferential NMOS input) for bitcount
values near zero, e.g., higher sup-
ply with 6-k LRS and lower supply
with 24-k LRS value.
The conventional scheme14
achieves energy efficiency of
20.8 TOPS/W at 1.2-V supply (see
Figure 6), while achieving 83.5%
accuracy with binary CNN. Jointly FIGURE 7. CIFAR-10 accuracy with voltage scaling for RRAM IMC with different LRS
optimizing the use of higher LRS values and input-splitting scheme.
value of 12 k (24 k ), the proposed
input-splitting scheme effectively
enabled voltage scaling down to 0.8 V without any addi- are a total of eight ADCs in the RRAM array periphery.
tional accuracy loss, improving the energy-efficiency The inference computation is processed layer by layer.
by 3.7× (5.2×), and achieving 80.9 (116.0) TOPS/W. Table 1 presents the benchmarking results with differ-
ent RRAM array sizes.
Two-Bit RRAM-Based CNN First, the inference accuracy drops as the array
Accelerator Performance size is increased, since the ADC precision is fixed at 5
Benchmarking with Neurosim bits. This is attributed to the fact that the partial sum
Two-bit RRAM could further increase the integration distribution becomes broader with larger array size,
density for the CNN accelerator. The performance and, therefore, quantization loss is increased. It should
benchmarking for 2-bit RRAM-based CNN accelera- be noted that the conductance variation of 2-bit RRAM
tor is conducted in NeuroSim,17 where the aforemen- only leads to small accuracy drop when comparing the
tioned VGG-like CNN is utilized. We assume that eight accuracy with ADC quantization only and with both
columns share one ADC in the RRAM array, and there ADC quantization and RRAM conductance variation.

www.computer.org/computingedge 35
EXPERT OPINION

TABLE 1. CNN simulation results with 2-bit RRAM for

different RRAM array sizes. CIFAR-10 data set. Experiments with 2-bit RRAM dem-
RRAM array size 64 64 128 128 256 256
onstrate sufficient separation between four conduc-
91.2% 1 89.2%1 79.0%1
tance levels, and show higher CNN accuracy up to 128
CIFAR-10 accuracy × 128 RRAM array size.
(91.4%) 2 (89.3%) 2 (81.2%)2

Chip area (mm 2 ) 19.64 19.21 14.71

Read dynamic energy
33.73 32.50
(layer-by-layer, J)
Leakage energy ( J) 1.78 0.77
Latency (ms) 8.90 6.11
Energy efficiency
17.35 18.52
(TOPS/W)
Throughput (FPS) 112.38 163.72
1 Accuracy with ADC quantization and RRAM conductance variation.
2 Accuracy with ADC quantization (without RRAM conductance variation).
In terms of chip area, 256 × 256 array shows smaller
chip area compared with 128 × 128 array due to the
increased array efficiency. However, only small chip
area increase is observed when array size is reduced
to 64 × 64. Comparing with 128 × 128 array, for 256 ×
256 array, chip area is reduced as less subarrays are
needed. It can be explained by the fact that in 64 × 64
array, the periphery circuit size is reduced due to lower
maximum column partial sum current, therefore, the
array efficiency does not drop significantly compared
with 128 × 128 array.
For the read latency and dynamic energy con-
sumption, comparing with 128 × 128 array, 64 × 64
array needs more partial sum accumulations between
subarrays, which leads to higher latency and energy
consumption. For 256 × 256 subarray, the large column
current leads to significantly higher ADC energy con-
sumption and, therefore, the overall energy consump-
tion is increased. Besides, the larger column partial
sum current leads to larger transmission gate (TG) size
in the multiplexer, which induces higher latency for the
decoder to drive the TG gate capacitor.
CONCLUSION
In this article, we demonstrated RRAM-based IMC
with 90-nm CMOS prototype chips that monolithically
integrated RRAM and CMOS in different vertical lay-
ers. Using device-/circuit-/algorithm-level techniques,
both the energy efficiency and density of binary
RRAM-based IMC hardware improved substantially,
achieving up to 116.0 TOPS/W and 84.2% accuracy for
ACKNOWLEDGMENT
55.46 The authors would like to thank Winbond Electron-
ics for chip fabrication support. This work was sup-
0.81
ported in part by NSF-SRC-E2CDA under Contract
12.93
2018-NC-2762B; by the National Science Founda-
10.95 tion (NSF) under Grant 1652866, Grant 1715443, and
Grant 1740225; in part by JUMP C-BRIC and ASCENT
77.36
programs (SRC programs sponsored by DARPA); and
in part by the Nano-Material Technology Develop-
ment Program through the National Research Foun-
dation of Korea funded by the Ministry of Science, ICT
(NRF-2016M3A7B4910249).
REFERENCES
1. Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,”
Nature, vol. 521, pp. 436–444, 2015.
2. X. Xu, et al., “Scaling for edge inference of deep neural
networks,” Nature Electron., vol. 1, pp. 216–222, 2018.
3. Z. Jiang, et al., “XNOR-SRAM: In-memory computing
SRAM macro for binary/ternary deep neural networks,”
in Proc. IEEE Symp. VLSI Technol., 2018, pp. 173–174.
4. H. Valavi, et al., “A 64-tile 2.4-Mb in-memory-computing
CNN accelerator employing charge-domain compute,”
IEEE J. Solid-State Circuits, vol. 54, pp. 1789–1799, Jun.
2019.
5. X. Si, et al., “A twin-8 T SRAM computation-in-memory
macro for multiple-bit CNN-based machine learning,” in
Proc. IEEE Int. Solid-State Circuit Conf., 2019, pp. 396–398.
6. V. Seshadri, et al., “Ambit: In-memory accelerator
for bulk bitwise operations using commodity DRAM
technology,” in Proc. IEEE/ACM Int. Symp. Microarchit.,
2017, pp. 273–287.
7. S. Li, et al., “DRISA: A DRAM-based reconfigurable
in-situ accelerator,” in Proc. IEEE/ACM Int. Symp.
Microarchit., 2017, pp. 288–301.
8. R. Mochida, et al., “A 4 M synapses integrated analog
ReRAM based 66.5 TOPS/W neural-network processor
with cell current controlled writing and flexible net-
work architecture,” in Proc. IEEE Symp. VLSI Technol.,
2018, pp. 175–176.
9. C.-X. Xue, et al., “A 1 Mb multibit ReRAM computing-in-
memory macro with 14.6 ns parallel MAC computing

36 ComputingEdge July 2021


time for CNN Based AI edge processors,” in Proc. IEEE
Int. Solid-State Circuit Conf., 2019, pp. 388–390.
10. M.-M. Shulaker, et al., “Three-dimensional integration
of nanotechnologies for computing and data storage
on a single chip,” Nature, vol. 547, pp. 74–78, 2017.
11. C. Ho, et al., “Integrated HfO2-RRAM to achieve highly
reliable, greener, faster, cost-effective, and scaled
devices,” in Proc. IEEE Int. Electron Devices Meeting,
2017, pp. 2.6.1–2.6.4.
12. I. Hubara, et al., “Binarized neural networks,” in Proc.
Adv. Neural Inf. Process. Syst., 2016.
13. X. Sun, et al., “XNOR-RRAM: A scalable and parallel
resistive synaptic architecture for binary neural
networks,” in Proc. Design, Autom. Test Eur. Conf.
Exhib., 2018, pp. 1423–1428.
14. S. Yin, et al., “High-throughput in-memory comput-
ing for binary deep neural networks with mono-
lithically integrated RRAM and 90 nm CMOS,” 2019,
arXiv:1909.07514. [Online]. Available: https://arxiv.org
/abs/1909.07514
15. Y. Kim, et al., “Input-splitting of large neural networks
for power-efficient accelerator with resistive crossbar
memory array,” in Proc. IEEE Int. Symp. Low Power
Electron. Design, 2018, Article no. 41.
16. S. Wu, et al., “Training and inference with integers in
deep neural networks,” 2018, arXiv:1802.04680. [Online].
Available: https://arxiv.org/abs/1802.04680
17. P. Chen, et al., “NeuroSim: A circuit-level macro model
for benchmarking neuro-inspired architectures in online
learning,” IEEE Trans. Comput.-Aided Design Integr.
Circuit Syst., vol. 37, no. 12, pp. 3067–3080, Dec. 2018.
SHIHUI YIN is currently working toward the PhD degree at
the School of Electrical, Computer and Energy Engineering,
Arizona State University. He is a student member of IEEE.
Contact him at: Shihui.Yin@asu.edu.
YANDONG LUO is currently working toward the PhD degree
at the School of Electrical and Computer Engineering, Geor-
gia Institute of Technology. He is a student member of IEEE.
Contact him at: yandongluo@gatech.edu.
YULHWA KIM is currently working toward the PhD degree at
the Department of Creative IT Engineering, Pohang Univer-
sity of Science and Technology. She is a student member of
IEEE. Contact her at: yulhwa.kim@postech.ac.kr.
www.computer.org/computingedge
EXPERT OPINION
WANGXIN HE is currently working toward the PhD degree at
the School of Electrical, Computer and Energy Engineering,
Arizona State University. He is a student member of IEEE.
Contact him at: wangxinh@asu.edu.
XU HAN is currently working toward the PhD degree at the
School of Electrical, Computer and Energy Engineering,
Arizona State University. She is a student member of IEEE.
Contact her at: xhan37@asu.edu.
XIAOYU SUN is currently working toward the PhD degree at
the School of Electrical and Computer Engineering, Georgia
Institute of Technology. He is a student member of IEEE. Con-
tact him at: xiaoyusun@gatech.edu.
HUGH BARNABY is a professor in the School of Electrical,
Computer and Energy Engineering, Arizona State University.
His research interests include device physics and modeling,
microelectronic device/sensor design and manufacturing,
and analog/RF/mixed-signal circuit design. He is a Fellow of
IEEE. Contact him at: hbarnaby@asu.edu.
JAE-JOON KIM is a professor in the Department of Creative
IT Engineering, Pohang University of Science and Technol-
ogy. His research interests include neuromorphic circuit and
system, low power VLSI design, and flexible device/circuit
design. He is a member of IEEE. Contact him at: jaejoon
@postech.ac.kr.
SHIMENG YU is an associate professor in the School of
Electrical and Computer Engineering, Georgia Institute
of Technology. His research interests are nanoelectronic
devices and circuits for energy-efficient computing systems.
He was a recipient of the NSF CAREER Award in 2016, the IEEE
Electron Devices Society (EDS) Early Career Award in 2017,
and the Semiconductor Research Corporation (SRC) Young
Faculty Award in 2019. He is a senior member of IEEE. Contact
him at: shimeng.yu@ece.gatech.edu.
JAE-SUN SEO is an assistant professor in the School of
Electrical, Computer and Energy Engineering, Arizona State
University. His research interests include energy-efficient
hardware design for machine learning and neuromorphic
computing. He received the IBM Outstanding Technical
Achieved Award in 2012 and the NSF CAREER Award in 2017.
He is a senior member of IEEE. Contact him at: jaesun.seo
@asu.edu.
37
 EDITOR: Ron Vetter, University of North Carolina Wilmington, vetterr@uncw.edu
DEPARTMENT: SPOTLIGHT ON TRANSACTIONS
Parallel and Distributed Systems
Manish Parashar, Rutgers University
This installment of Computer’s series highlighting the work published in IEEE Computer
Society journals comes from IEEE Transactions on Parallel and Distributed Systems.
P
arallel and distributed computing systems
have made significant contributions to the
advancement of machine learning. The recent
success of machine learning technologies is due not
only to new algorithms that improve accuracy but also
to new algorithms and systems that exploit special-
ized high-performance hardware [for example, graph-
ics processing units (GPUs) and field-programmable
gate arrays] to improve efficiency. This research topic
has been even more important in the era of big data.
Gradient boosting decision trees (GBDTs) have been
widely used in advertising systems, spam filtering,
sales prediction, medical data analysis, and image
labeling. In recent years, they have become very
popular and won many awards in machine learning
and data mining competitions.
In “Exploiting GPUs for Efficient Gradient Boost-
ing Decision Tree Training,”1 Wen et al. present a
series of novel optimization techniques on GPUs
for accelerating GBDT training tenfold over their
CPU counterparts and improving scalability on
high-dimensional data over their GPU equivalents
(see Figure 1). Accelerating GBDT training on GPUs
is fundamentally challenging due to the large
number of irregular memory accesses required by
the tree structures and the need for frequent data
partitioning.
Digital Object Identifier 10.1109/MC.2020.3017320
Date of current version: 21 October 2020
38 July 2021 Published by the IEEE Computer Society 
This article originally
appeared in
vol. 53, no. 11, 2020
The solution proposed by the authors thoroughly
optimizes various aspects of the training, including 1)
data compression using run-length encoding, 2) fast
data partitioning using stable sort, 3) approximation
in finding the split for a node using two-stage histo-
gram building, 4) building histograms that are aware
of data sparsity, 5) reusing intermediate result during
training, and 6) exploiting multiple GPUs to handle
larger data sets. These techniques, taken together,
GRADIENT BOOSTING DECISION
TREES HAVE BEEN WIDELY USED
IN ADVERTISING SYSTEMS, SPAM
FILTERING, SALES PREDICTION,
MEDICAL DATA ANALYSIS, AND
IMAGE LABELING.
vastly increase the performance and scalability of
GBDT training on GPUs and form the critical founda-
tions of an open sourced system in GitHub named
ThunderGBM. Comprehensive experimental results
on popular data sets confirm the effectiveness of
ThunderGBM over the existing GBDT implementa-
tions, including XGBoost, LightGBM, and CatBoost on
both GPUs and CPUs.
A lthough accelerating the efficiency of machine
learning systems on GPUs is not new, irregular
2469-7087/21 © 2021 IEEE
 xgboost-cpu catboost-cpu
10 3
lightgbm-cpu thundergbm

10 2
Time Elapsed (s)

2
n/a

n/a

n/a
n/a
0
e

im
s

sy
6

p
yp

in
gg

s2
00

g1
su
-s
vt

hi

lo
e2

al
ne
co

re

FIGURE 1. Comparison with existing libraries on the GPU and CPUs.1

accesses and frequent data partitioning make GBDT MANISH PARASHAR is the Distinguished Professor of
training different and more challenging when com- Computer Science at Rutgers, The State University of New
pared to regular training systems. The authors have Jersey. He is the editor in chief of IEEE Transactions on Paral-
addressed these technical challenges with novel and lel and Distributed. Contact him at parashar@rutgers.edu.
efficient solutions as well as integrated the tech-
niques into an easy-to-use system. The promising
speedup achieved through GPU acceleration can pro-
mote further research on accelerating other machine
learning systems with irregular access patterns (such
as sparse networks). On the other hand, the success
of this project also increases the feasibility of more
real-time applications such, as fraud detections in
future digital finances using GBDTs.

REFERENCE
1. Z. Wen , J. Shi, B. He, J. Chen , K. Ramamohanarao, and
Q. Li, “ Exploiting GPUs for efficient gradient boosting
WWW.COMPUTER.ORG/COMPUTINGEDGE
decision tree training ,” IEEE Trans. Parallel Distrib.
Syst., vol. 30, no. 12 , pp. 2706 –2717, 2019.

www.computer.org/computingedge 39
 EDITORS: Phil Laplante, plaplante@psu.edu
Ben Amaba, baamaba@us.ibm.com
DEPARTMENT: INTERNET OF THINGS
To Err is Human, to Forgive, AI
Phil Laplante, Penn State
Ben Amaba, IBM
T
rustworthiness is an elusive quality. We may
completely or partially trust relatives, friends,
colleagues, or strangers. We also place a great
deal of trust in the operators of airplanes, cars, medi-
cal prognoses, invasive medical devices, and other
complex systems, potentially risking our lives doing
so. Similarly, we trust that the designers, builders,
testers, operators, and maintainers of these com-
plex systems took great care in ensuring safety and
reliability. But no matter what measures are taken to
ensure error free operation, we acknowledge a cer-
tain level of risk of failure, even catastrophic failure
in these systems, because they are built and oper-
ated by humans.
What about those systems that employ artificial
intelligence (AI), such as driverless cars, autopilots,
invasive medical devices, and certain types of sys-
tems in the internet of things? Do we expect these
AI enabled systems to operate in such a way that
they can be trusted more than those that are oper-
ated only by humans? It seems to be headline news
when an AI capable system fails, particularly when
the blame can be placed directly on the underlying
“intelligence.” But we should not be surprised when
AI enabled systems fail and we can prove it to you?
In fact, while we strongly advocate for such systems,
we think we should insist on an even higher level
of professionalism and rigor when developing and
deploying AI systems.
Digital Object Identifier 10.1109/MITP.2019.2913265
Date of current version 17 July 2019.
40 July 2021 Published by the IEEE Computer Society 
This article originally
appeared in
vol. 21, no. 4, 2019
WHO CAN YOU TRUST?
Trust in an entity (either human or a system) is con-
stantly evolving, but there is a base trust established
at first encounter. We all have a certain “trust toler-
ance profile” based on individual factors such as
upbringing and life experience. For example, some
people have a very low initial trust tolerance and tend
to distrust people upon the first encounter. Other
people, however, may trust the same person substan-
tially from the outset.
An initial trust level is also established for a
system based on its source, certifications, licenses,
regulatory guidelines, operating history, and/or on
the user's trust profile. For example, you may not
want to be the first person to ride in a newly released
autonomous vehicle but have friends who would
leap at the chance. Who is right here? To answer, let
us consider the initial encounter with any generic AI
enabled system.
During a crisis of faith, while living a monastic life,
mathematician Blaise Pascal used expected value
theory to conclude that it was better to believe in a
divine being than not. We now call this formulation
“Pascal's Wager” and a sort of complementary analy-
sis can help us answer the question of whether to
trust any AI enabled system upon the first encounter.
Consider the confusion matrix in Figure 1.
A human can either trust the AI system or not.
If the AI system is trustworthy, the consequences
of the interaction are positive. If the human trusts
an untrustworthy AI system, however, then the con-
sequences can be extremely negative (for example,
a robotic surgical device making a fatal error). But
when the human already distrusts the AI system,
2469-7087/21 © 2021 IEEE

FIGURE 1. Confusion matrix for human-AI prisoner’s dilemma problem.
then presumably, precautions would have been taken
for the consequences of failure, making them less
severe, for example, having some sort of human over-
ride mechanism when the system goes awry. Finally,
distrusting a trustworthy AI system may have some
slight cost, for example, extraneous error checking,
marginally decreasing the maximum benefit of the
interaction. Roughly speaking, the formulation of
the consequences is represented by the following
equations:
Trust AI :V(consequences)
= P(AITrustworthy) * V(positiveconsequences)
+ P(AInotTrustworthy) * V(verynegativeconsequences)
(1)
Distrust AI: V (consequences)=P(AITrustworthy) *
V (marginallypositiveconsequences)
+ P(AInotTrustworthy)*
V (marginally negative consequences)(2)
The function P() represents “the probability of,”
and V() represents “value of,” whether in financial or
costs (e.g., human lives, injury). We can tinker with the
probabilities and the costs consequences to make
it seem better to trust the AI, but in safety-critical
systems, the cost of failure is always going to be
extremely high (e.g., death, serious injury, destruction
of costly infrastructure). Thus, if the decision to trust
is based strictly on minimizing the maximum loss, and
not on some personality-based objective, such as the
thrill seeking, then it is always better to distrust the AI
system on the first encounter.
MONITORING THE ELUSIVE
TRUST FACTOR
Over time, our trust in another person or a system
evolves through direct experiences and other evi-
dence such as third-party reports, news, and rumors.
www.computer.org/computingedge
INTERNET OF THINGS
For both human and AI-enabled system, we are con-
stantly reevaluating our trust in that entity. Should
we ever trust any entity completely, even after a
long history of trustworthy interactions? We think
not. Let us look at a couple of models to justify this
sentiment.
Using Oracles
For every system controlled by AI (or, for some human),
we might think we could invent some kind of oracle to
track the response to each interaction. For each set of
stimuli, the system acts like a filter, producing a set of
response. Generically, the system acts like an if-then
statement, i.e, the following:
if (set of stimuli) then (set of responses).
We then evaluate if the responses are trustwor-
thy, that is, as expected, not expected, or forbidden.
These behaviors can, theoretically, at least, be speci-
fied in a tabular form, which helps us predict (and
trust) the transactions, perhaps allowing us to adjust
the probabilities in (1) and (2). But by creating this,
the table is just a version of the Gödel incomplete-
ness problem, and it can be shown, using Cantor's
Diagonal argument, that the table can never be com-
pleted. That is, there is always a new, unencountered
transaction, which should not be trusted in the sense
of Pascal's Wager.
Interrogating Prisoners
In the absence of an oracle to determine the trust of
an AI system, could a probative mechanism be used?
A common technique for interrogating prisoners of
war can be used to continuously test the trustwor-
thiness of humans and AI systems. The interrogator
asks the prisoner a series of five to ten questions of
which the truthful answer is known. If the prisoner
truthfully answers the first n questions, then there
41
INTERNET OF THINGS
is a certain degree of confidence in the response to
next questions (with unknown answers). In fact, as
the number of consecutive truthful answers given
increases, the likelihood of the unknown question
being false diminishes.
In software testing, Miller et al.1 showed that
given T consecutive tests without a failure:
P (FailontestT+1afterTsuccessfultest) = 1/(n + 2). (3)
By simply restating the problem as a trust test
for the AI system and framing the “questions” as a
set of stimuli to which we expect a response, we can
deduce that the probability of betrayal on the (n +
1)st interaction (after n trusted interactions) is 1/(n
+ 2). So, as n grows large, the likelihood of betrayal
tends towards zero (unless we pass to the limit of
infinity, which is impossible). But we can use (3) to
update the probabilities that an interaction has a
trustworthy answer in (1) and (2). What this all means
is that we need to continuously and exhaustively
test AI-based systems in order to maintain a reason-
able level of trust. But we can never completely trust
the system.
TO ERR IS HUMAN
In his 1942 science fiction story “Runaround,” Isaac
Asimov proposed three laws that must govern the
behavior of robots:
R1. A robot may not injure a human being or,
through inaction, allow a human being to come
to harm.
R2. A robot must obey any orders given to it by
human beings, except where such orders would
conflict with the First Law.
R3. A robot must protect its own existence as long
as such protection does not conflict with the
First or Second Law [Asimov].
Asimov later added a 0th law:
R0. A robot may not harm humanity, or, by inaction,
allow humanity to come to harm.
This seems like a reasonable framework for AI deci-
sion that could promote trust. But the framework can
42 ComputingEdge
lead to all kinds of moral dilemmas, such as an intel-
ligent car having to decide between running off the
road and crashing into a school bus full of children, an
autopilot system choosing where to crash to minimize
causalities, or the classic “trolley problem.” We have
no way to quantify the subtle considerations that
have to be made in ethical dilemmas and, hence. do
not know how to build such AI. Furthermore, Asimov's
laws do not inherently account for the possibility that
the system will fail, that is, deliver imperfect data or
make an imperfect decision.
Finally, any AI system that would choose to destroy
itself rather than harm another would not pass the
Turing test. The philosopher Spinoza said “What if a
man could save himself from the present danger of
death by treachery? … If reason should recommend
that, it would recommend it to all men …” Consider
the computer (HAL) in the movie 2001: A Space Odys-
sey (1968). HAL, an AI system that is “foolproof and
incapable of error,” malfunctions, and, before it can be
taken offline, deliberately kills one of the astronauts to
save itself. An AI system that had faithfully executed
Asimov's Laws would not attempt this murder, but a
truly “intelligent” system trying to save itself would, as
would a faulty AI system.
TO FORGIVE, AI
Even if systems that behave like they should be per-
ceived as untrustworthy when some party sees the
outcome as unfair, we expect humans to make mis-
takes, to disappoint us by marginal performance, and
even to possibly betray us. But should we expect per-
fection from AI systems or forgive them when they do
fail? True AI means that the system must have a cer-
tain probability of acting in an aberrant (untrusted)
manner. We can either build in the potential for
betrayal or expect it, but in either case, we have to
deal with the possibility in the outcome space.
I am not suggesting that AI-enabled systems are
bad. On the contrary, they can make our lives easier
and better. It is just that we should not trust them
out-of-the-box to be any safer than non-AI-enabled
systems. We must insist that they are developed in
a very scrupulous and professional manner by those
who really know what they are doing, and we must
insist that they prove their trustworthiness con-
tinuously. According to poet Alexander Pope “To err
July 2021
 INTERNET OF THINGS

is human, to forgive, divine.” But if he knew about AI,

Cutting Edge
he would have amended this to “To err is human, to
forgive, divine—but don't trust AI.” stay
on the
REFERENCES
1. K. Miller, et al., “Estimating the probability of failure
when testing reveals no failures,” IEEE Trans. Softw.
of Artificial Intelligence
Eng., vol. 18 , no. 1, pp. 33 – 43, Jan. 1992.
J a n ua ry/ f E b r ua ry 2 016

IEEE Intelligent Systems provides peer-

IEEE
2. I. Asimov, “Runaround,” Astounding Sci. Fiction, vol. 29, Also in this issue:
aI’s 10 to Watch
real-Time Taxi Dispatching
56
68

IEEE
January/FEBruary 2016
from flu Trends to Cybersecurity 84

no. 1, 1942. P U T T I N G A I I N T O P R A C T I C E

reviewed, cutting-edge articles on the

theory and applications of systems

Online BehAviOrAl AnAlysis

that perceive, reason, learn, and
PHIL LAPLANTE is a Professor of software and systems
act intelligently.

VOLuME 31
engineering with the Pennsylvania State University, Univer-

nuMBEr 1
www.computer.org/intelligent

IS-31-01-C1 Cover-1 January 11, 2016 6:06 PM

sity Park, PA, USA. Contact him at plaplante@psu.edu.

BEN AMABA is a Chief Innovation Officer for the Industrial

Manufacturing and Engineering Sector and a member of
The #1 AI Magazine
www.computer.org/intelligent

IEEE
the Artificial Intelligence and Data Science Elite Team, IBM,
Armonk, NY, USA. Contact him at baamaba@us.ibm.com

Call rticles
for A ing
e C o mput
iv
EE Pervas o n th
e late
st
IE , u s ef
ul p a p e r s
e,
a c ce s s
ible r vasiv
seek s e nt s in pe
v elopm ics
e d de g. Top
eview putin
peer-r ous co
m
u biquit a re
e , an d , s of t w
mobil e c h n ology
t
ware an d
e hard ensing
includ w orld s
l-
re , re a c tion,
f ra s truc tu u t e r intera
s: in mp
e li n e an- co
or gu
id
c tio n , hu m includ
ing
Au t h /mc / inter a
tions,
er.org e ra
privac
y.
mpu t consid
ww w .c o
htm d s y s te m s c u r it y, and
uthor. an se
sive /a bilit y,
p e r v a
ils: m e n t, scala
a y
e r de t deplo
Furth o rg
puter. sive
sive @ co m
g /perva
p e r va ter.or
.c ompu
www

www.computer.org/computingedge 43
 EDITOR: Dimitrios Serpanos, ISI/ATHENA and University of Patras, serpanos@computer.org

This article originally

appeared in
DEPARTMENT: CYBER-PHYSICAL SYSTEMS
vol. 53, no. 1, 2020

Flexibility in Production Systems

by Exploiting Cyberphysical Systems
Arndt Lüder, Otto-von-Guericke University, Magdeburg

New IT and computer science solutions are needed to ensure an increased flexibility of
production systems. One key is using intelligent and self-responsible production system
components, the Industry 4.0 components. In this column, relevant requirements,
research and development trends, and issues still to be addressed are presented.

P
roduction systems, in general, are charac-
terized by the three main concepts they
combine: products, production resources,
and production processes (Figure 1).1 The purpose
of a production system is to produce products in an
amount and quality to meet demands of possible cus-
tomers. The product’s design defines the appearance
and functioning of the final products by combining
and processing the defined materials in an estab-
lished way. The main outcome of the product design,
from the production system point of view, is the set of
required materials (the bill of material definition) and
a partially ordered sequence of production processes
to be executed on them (the bill of operation).
Production resources are used to ensure the
implementation of the processes. The resources are
organized in a hierarchy of production system compo-
nents, whereby each component executes a share of
the production process provided by the resource. The
complexity of this share can range from a very simple
task, like opening a valve or rotating a drive, to com-
plex processing, like welding a line. In general, each
component provides production functionalities that
add value to the product or support the value-adding
Digital Object Identifier 10.1109/MC.2019.2949107
Date of current version: 15 January 2020
processes that enhance the production skills of the
resource.
The connection between products and resources
is established by the production processes that are
required to create the products and are provided by
the resources. In traditional production systems, the
portfolio of possible products is strongly interlinked
with the set of resources that are used. This linking
is done when the production system is engineered
by integrating the production process characteris-
tics required for the intended products within the
structure of the production components (technical
bordering conditions) and within the control software
(sequence and parameterization of necessary process
steps). Thus, only those products that are predicted in
the design stage can be created within such produc-
tion systems.2
Increasing competition in global markets is leading
to new challenges. Customer requirements are chang-
ing very fast. Technological process is driving changes
related to applicable production system components.
Lawmakers are increasingly facing social and ecologi-
cal challenges that increase the number of regulations
to be considered. All of these challenges change (in
fact reduce) the time that classically designed produc-
tion systems are economically viable.
In response, a new type of production system is
being considered,3 based on self-reliable production

44 July 2021 Published by the IEEE Computer Society  2469-7087/21 © 2021 IEEE
 Is Scheduled On
Product Resource
Is Producing

Can Make Shall Execute

Process
Requires Can Execute

FIGURE 1. Relations among products, production processes, and production resources.

system resources established by a hierarchy of as technical, quality related, economic, and

self-aware production system components and prod-
ucts. Within such a production system, products will
be able to automatically find the resources needed
to execute the production processes required to cre-
ate them. Thus, the process of linking product and
production resources is done while the production
system is operating.
To make such a production system applicable, pro-
duction resources need to
›› integrate themselves in the overall production
system through their own ability to plan and
address their own maintenance requirements
›› express their skills, including those related
to technical requirements, quality, economic
benefits, and ecological issues
›› expose interfaces that enable other entities in
the production system to use the resources.
In addition, products need to
›› know details about the way they have to be
produced, including sequence and param-
eterization of necessary process steps as well
ecological bordering issues
›› be able to use the interfaces exposed by
resources.
PRODUCTION SYSTEM CONTROL
Traditionally, control of production systems follows
the pattern of the classical automation pyramid,4
which usually establishes a hierarchy of production
system components controlled in a closed-loop struc-
ture. This closed loop is established by measuring the
state of the technical system using appropriate sen-
sors, deciding on necessary control actions within
controllers based on measured state and behavior
specifications, and executing these control actions
by actors. Thus, information processing is essential
for guiding the behavior of the production system.
The controllers within these closed loops estab-
lish a distribution of necessary control decisions. 5
According to the classical approach of the automation
pyramid, this distribution forms a strict separation
of considerations where lower-layer controllers are
responsible for the behavior of smaller components,
while higher-layer controllers control larger compo-
nents.1 So lower-layer controllers are in a position to

www.computer.org/computingedge 45
CYBER-PHYSICAL SYSTEMS
Administration Shell 2 The envisioned Industry 4.0 components and their use
4
Technical Virtual
3 Functions Representation
6 1
5 well as cooperation. Several research and develop-
Controller
Thing
FIGURE 2. The architecture of the Industry 4.0 component
with an asset administration shell.
more closely control production system physics than
higher-layer controllers, which are specifically respon-
sible for planning decisions.
Depending on industry, bordering conditions of
the production system, intended system structuring,
different types of control devices, and control soft-
ware are applied. These systems are programmable
logic controllers, numeric controllers, robot control-
lers, PC-based controllers, and others with their
specialized languages but also with programming
languages like C, Python, or (seldom) Java.
To accommodate greater flexibility, the auto-
mation pyramid is increasingly giving way to more
heterarchical and distributed control architectures
that combine lower-layer (process control) and
higher-layer control (planning) decisions within one
device related to one production system component,
leading to so-called cyberphysical systems (CPSs).
According to Encyclopedia of Business Informatics,
a CPS is a combination of components with physical
and data processing parts involving communication.
CPSs range from very simple embedded systems, for
example, drives or rotary encoders, to very large ones,
such as production systems, power plants, energy
transmission systems, airplanes, and train systems.
Cyberphysical production systems are intended
to be used to form the new type of production sys-
tems already described. But to do so, they have to ful-
fill the requirements presented for production system
components.
46 ComputingEdge
THE INDUSTRY 4.0 COMPONENT
within Industry 4.0 production systems will provide the
solution to the previously mentioned challenges. They
will employ CPSs with capabilities for self-awareness,
self-planning, self-control, and self-maintenance as
ment activities are moving in this direction in Germany
and elsewhere. The results are interesting, but several
issues remain open.
Reached developments
The different ongoing research and development
activities naturally have various goals. These range
from developing general architectures and detailed
designs of architecture components to proving the
applicability of different technologies within the over-
all architecture or their components. Handbuch Indus-
trie 4.0 Bd. 2: Automatisierung6 includes a collection
of application cases and descriptions of technol-
ogy descriptions. The “Industry 4.0 Platform”7 shows
results of standardization efforts. Both sources are
regularly updated as progress is made.
All research and development activities employ a
common general architecture pattern for the Indus-
try 4.0 component depicted in Figure 2.7 This com-
ponent is an identifiable, communication-enabled,
and cooperative combination of one or more
physical assets accompanied and supervised by
an administration shell.7 This shell provides a vir-
tual representation of the covered asset as well as
access to its technical functionalities. The virtual
representation can be viewed as a semantically and
metadata-enriched collection of the possibly inter-
linked individual sets of data describing the asset
from all perspectives, which might be relevant within
its lifecycle.
The physical asset is not necessarily a simple or
trivial object. It can contain its own controller, pro-
viding it with all necessary intelligence and control
decisions for the execution of the specified technical
functions. Thus, two intelligent pieces may be involved
in the Industry 4.0 component: the management shell
and the asset internal control.
This architecture calls for implementation deci-
sions addressing six problems. First, the virtual
representation has to be set up [(1) in Figure 2]. Here,
July 2021

the different lifecycle phases of the asset need to be
considered with respect to relevant data sets and
usable implementation technologies. While data for-
mats such as AutomationML are considered during
engineering, Open Platform Communications (OPC)
Unified Architecture (UA) is the top consideration
during use. Between the two phases, that is, during
system setup, specialized XML dialects or Java Script
Object Notation can become relevant. The data struc-
turing in the virtual representation is currently under
development.7
The second problem is the realization of the access
to the virtual representation [(2) in Figure 2]. Here, the
implementation technologies of the virtual represen-
tation define possible access paths. Especially during
use, OPC UA gives detailed implementation specifica-
tions. During engineering, this is more open. Depend-
ing on the engineering network architecture, different
types of information logistics implementations can be
applied.8
The third problem involves implementation of the
supervision and provision of the technical functions
[(3) in Figure 2]. Currently, the use of service-oriented
architectures is being explored. This technology
enables the self-description of services and their
orchestration, following the needs of the accessing
entity.
Consequently, access to the technical functions
[(4) in Figure 2] is also realized by service access. It
enables the use of existing IT technologies as well as
smooth integration with existing IT landscapes.
The integration of the technical function realiza-
tion within the asset with its supervision and control
within the administration shell [(5) in Figure 2] is
highly asset dependent. Nevertheless, OPC UA is
one of the most commonly considered technologies
for this integration. Within the German Mechani-
cal Engineering Industry Association, several
industry-driven OPC UA compendium specifications
are under development for that purpose.
The last problem is the integration of the super-
vision and control of the technical functions with
the virtual representation [(6) in Figure 2]. The main
purpose of this integration is to enable the provision
of semantically enriched asset state and behavior
information accompanied by engineering data.
OPC UA is one candidate for the realization as well,
www.computer.org/computingedge
CYBER-PHYSICAL SYSTEMS
independently of the realization of the virtual repre-
sentation that may be with OPC UA or AutomationML.
The current state of the art enables
resource-related flexibility. Industry 4.0-based pro-
duction system components can integrate them-
selves in the overall production system. They can
expose interfaces that enable their use by other
production system entities and plan their actions, if
requested from outside, based on described skills.
Thus, a high degree of resource and process flex-
ibility is reached.
TO ENABLE INCREASED PROCESS
FLEXIBILITY THROUGH THE
OPTIMIZED USE OF INDUSTRY
4.0 COMPONENTS, THEIR SELF-
DESCRIPTION NEEDS TO BE
IMPROVED.
Open issues
There remain several open questions to increase
product, process, and resource flexibility within pro-
duction systems while also fulfilling all of these var-
ious requirements. To enable complete self-control
of production resources including self-planning,
self-maintenance, and similar capabilities and, thus,
increase resource flexibility, Industry 4.0 components
cannot be only service oriented and, thereby, reactive;
they need to be proactive. Agent-based implementa-
tions that extend service orientation provide a prom-
ising solution.9
To enable increased process flexibility through
the optimized use of Industry 4.0 components, their
self-description needs to be improved. On one hand,
the description of technical functions needs to be
detailed. The current focus on technical border-
ing conditions needs to be enlarged to integrate
reachable production process quality, economic
effects, ecological effects, and so on. Therefore,
new methods of skill representation and evaluation
are required. At the same time, the representation of
data within the virtual representation of the Industry
4.0 component needs to be standardized. This stan-
dardization will not only define the metastructure
47
CYBER-PHYSICAL SYSTEMS
of the data but also specify how the different related
engineering and usage disciplines will be represented
and integrated.
Additionally, the product has to be considered
an independent and proactive Industry 4.0 compo-
nent, driving the production process and increasing
product flexibility. Agent technologies, which have
been considered for several years as a means for
distributed proactive control, can assist the imple-
mentation. However, a slightly different component
architecture may be required to address the special
requirements of product flow through in a produc-
tion system.
T he state-of-the-art related to Industry 4.0-based
flexibility enables a wide range of successful appli-
cations that provide increased gains within industry.6
There are still several steps to take on the way to pow-
erful industrial solutions that fulfill the requirements of
Industry 4.0. We will make them based on established
technologies like Jade, OPC UA, and AutomationML
on their improvement, especially with respect to their
required integration, and on their enhancement with
completely new ideas like automatic self-generation
based on engineering data or automatic update fol-
lowing plug-and-play methods.
REFERENCES
1. S. Biffl, A. Lüder, and D. Gerhard, Eds.,
Multi-Disciplinary Engineering for Cyber-Physical
Production Systems: Data Models and Software Solu-
tions for Handling Complex Engineering Projects. New
York: Springer-Verlag, 2017.
2. H. ElMaraghy, “Flexible and reconfigurable manufac-
turing systems paradigms,” Int. J. Flexible Manuf. Syst.,
vol. 17, no. 4, pp. 261–276, 2005.
3. H. Kagermann, W. Wahlster, and J. Helbig, Eds., Umset-
zungsempfehlungen für das Zukunftsprojekt Industrie
4.0: Abschlussbericht des Arbeitskreises Industrie 4.0,
Bundesministerium für Bildung und Forschung. Frank-
furt am Main, Germany: Forschungsunion Wirtschaft
und Wissenschaft, 2013.
4. E. Trunzer et al., “System architectures for Industrie
4.0 applications: Derivation of a generic architecture
proposal,” Prod. Eng., vol. 13, nos. 3–4, pp. 247–257, June
2019. doi: 10.1007/s11740-019-00902-6.
5. T. Wagner, C. Haußner, J. Elger, U. Löwen, and A. Lüder,
48 ComputingEdge
“Engineering processes for decentralized factory
automation systems,” in Factory Automation, J.
Silvestre-Blanes, Ed., Vienna, Austria: In-Tech, 2010.
[Online]. Available: http://www.intechopen.com
/articles/show/title/engineering-processes-for
-decentralized-factory-automation-systems
6. B. Vogel-Heuser, T. Bauernhansel, and M. ten Hompel,
Eds., Handbuch Industrie 4.0 Bd.2: Automatisier-
ung, Berlin, Germany: Springer-Verlag, 2017. doi:
10.1007/978-3-662-53248-5.
7. Federal Ministry for Economic Affairs and Federal
Ministry of Education and Research. “Industry 4.0
Platform.” Accessed on: 2019. [Online]. Available: https:
//www.plattform-i40.de/PI40/Navigation/DE/In-der
-Praxis/Online-Bibliothek/online-bibliothek.html
8. A. Lüder, J.-L. Pauly, F. Rinker, and S. Biffl, “Data
exchange logistics in engineering networks exploiting
automated data integration,” in Proc. 24th IEEE Conf.
Emerging Technologies and Factory Automation,
Zaragoza, Spain, Sept. 2019. doi: 10.1109/ETFA
.2019.8869352.
9. VDE-GMA, Frankfurt am Main, Germany, “Status
report: Agents for the realization of Industrie 4.0,” Aug.
2019. [Online]. Available: https://www.vdi.de/ueber
-uns/presse/publikationen/details/agents-for-the
-realisation-of-industry-40
ARNDT LÜDER is the head of the Institute of Ergonomics,
Manufacturing Systems and Automation, Otto-von-Guericke
University, Magdeburg, Germany. He is a Member of the IEEE
Industrial Electronics Society and a member of the board of
directors of the AutomationML Association. Contact him at
arndt.lueder@ovgu.de.
F O LLOW US
@ s e cu rit y p riv a c y
July 2021
 EDITOR: Robert Blumen, SalesForce, robert@robertblumen.com
This article originally
appeared in
DEPARTMENT: SOFTWARE
ENGINEERING RADIO vol. 37, no. 5, 2020

Bert Hubert on the Domain

Name System
Gavin Henry

FROM THE EDITOR

Bert Hubert, author of the open source PowerDNS nameserver, discusses Domain Name System (DNS)
security and all aspects of DNS. Host Gavin Henry spoke with Hubert about what DNS is as well as
its history, attacks, flaws, privacy, encryption, integrity, trust, defending against query failure, DNS
over HTTPS, DNS over Transport Layer Security (TLS), DNS Security Extension (DNSSEC), HTTP/2
with DNS, Quick User Datagram Protocol (UDP) Internet Connections, monitoring DNS traffic, cache
poisoning, amplification attacks, UDP attacks, URL fetch attacks via Twitter, browser DNS lookups,
social-engineering attacks, and what you need to worry about DNS as a software engineer. We provide
summary excerpts below; to hear the full interview, visit http://www.se-radio.net or access our archives
via RSS at http://feeds.feedburner.com/se-radio.—Robert Blumen

Gavin Henry: What is a DNS? What else is stored in a DNS?

Bert Hubert: When you type a name in the Internet,
the computer cannot directly connect. It must look
up the Internet Protocol (IP) address of that website—
IPV4 or IPV6—which DNS provides. A lot happens to
make DNS work well. Browsers ensure that their que-
ries get answers quickly, because speed affects the
user experience.
It’s an old protocol that we rely on, and it’s tricky.
There is a difference between “this name doesn’t
exist” and “the name exists, but it doesn’t have an IPV6
address.” The latter says the name does exist, but the
thing you asked for doesn’t. But often, you get back
the answer, “the name doesn’t exist,” or “we tried to
resolve this name, but we it didn’t work.” This has led
to websites disappearing from the web because a load
balancer messed up this nuance.
Digital Object Identifier 10.1109/MS.2020.3000883
Date of current version: 20 August 2020
DNS also denotes where the mail server for a domain is
located, as an example. That’s the mail exchanger (MX)
record. If you have a network with many subscribers
and you’re worrying about bad behavior, such as serv-
ers that have been hacked, you can check in DNS how
many MX domain queries the server is doing because
hacked servers will attempt to spam widely. Then, you
see tons of MX records.
More frequent than MX records are text records
that store arbitrary bits of text in DNS. These are used
by many spam lists to identify IP addresses known to
be spamming or domain names known to be phishing.
Many mail servers perform DNS text-record lookups to
figure out if a sending mail server is known to spam.
With all its faults and its age, DNS is still the one tech-
nology that functions as a low-investment worldwide
distributed database; you can ask it many questions
per second and get good answers.
What types of DNS servers are there?

2469-7087/21 © 2021 IEEE Published by the IEEE Computer Society July 2021 49
SOFTWARE ENGINEERING RADIO
SOFTWARE
ENGINEERING RADIO
Visit www.se-radio.net to listen to these and other
insightful hour-long podcasts.
RECENT EPISODES
»» 413—Spencer Kimball, cofounder and CEO
of Cockroach Labs, talks with host Akshay
Manchale about CockroachDB, an open source
distributed database system.
»» 412—Host Felienne speaks with Gavis-Hugh-
son about how to prepare for the dreaded
“whiteboard interview,” and how to get better at
doing interviews.
»» 409—Joe Kutner, software architect for Heroku
at salesforce.com, discusses the 12-factor app
with host Kanchan Shringi.
UPCOMING EPISODES
»» Adam Shostack discusses threat modeling with
host Justin Beyer.
»» Host Adam Bell talks about incident response
with Berkay Mollamustafaoglu.
»» Vladimir Khorikov discusses functional pro-
gramming in enterprise applications with host
Jeremy Jung.
There are authoritative name servers—those that
say, “I know all about some website, and this is the
IP address of that website.” Queries on the Internet
start with root servers: more than 1,000 servers with
26 IP addresses. You can query these 26 IP addresses
and get an answer. Root servers don’t know a lot,
but they can give you a list of IP addresses for the
dot-com servers. You can query the dot-com serv-
ers for an IP address, and a dot-com server will say, “I
don’t know, but I do know where the name servers are
for that website. Ask over there, and here are their IP
addresses.” And then you end up with this authorita-
tive server that I first mentioned, and that server can
actually tell you the right IP address.
50 ComputingEdge
This process was originally thought to be too
much work for simple computers. So a separate server
was invented called a resolver (or a cache), which
would traverse these authoritative servers, start-
ing from the root server, to find the answer to your
question. Thus, in your browser, you type the domain
name you want to visit. The browser sends the query
to its configured resolver that is part of your local
network. And the resolver trolls the whole Internet
and consults authoritative servers until it finds the
right answer.
The resolvers come with a hints file that contains a
list of about 26 IP addresses, which might be outdated
if you haven’t turned the computer on for a few years.
At startup, a resolving name server tries these hints
records to find a name server that answers. Chances
are that the resolver will find one IP address that
works, and from there, it can learn the latest list of 26
IP addresses.
What about integrity and encryption with a DNS?
DNSSEC adds integrity to a DNS; if you have an answer,
it can validate the answer. But applications that use a
DNS typically don’t validate DNS answers. Even brows-
ers simply query a name server and trust the answer.
It’s technically possible for a browser to validate DNS
answers using DNSSEC, and many domain names are
now DNSSEC signed. But browsers have decided to
put trust in TLS certificates instead.
TLS provides transport security. It validates that
the data you received are what you should have
received. DNSSEC works more like Pretty Good Pri-
vacy or secure Multipurpose Internet Mail Extension
programs—it signs the final answer to your question,
which is the actual name/IP-address combination.
That has made DNSSEC difficult to implement.
The idea behind TLS is that you trust the certifi-
cate’s final verification of HTTPS. If you trust HTTPS for
message integrity or transport integrity, you no longer
care about the IP address because the transport has
integrity. The common view is that you don’t need to
worry about DNSSEC if you’re encrypting. But there
have been brief hijacks of the DNS name in which
people have been able to get new TLS certificates dur-
ing the hijack. That is where DNSSEC validation would
have helped.
July 2021
 SOFTWARE ENGINEERING RADIO

What about privacy?
DNS traffic tells you everything about a person. If you
have access only to someone’s DNS records, you will
be able to tell where they live, what phone they have,
or what brand of TV they have, and so forth. Per bit,
DNS may be the most privacy-sensitive material on
the Internet. It is worthwhile to protect DNS records
because we don’t want too many people to have
access to those.
If someone has a Tesla and I know that they work
at a certain company and which sports team they
like, that narrows down a search for a person. All this
information radiates from that person’s Internet con-
nection through a DNS.
A common current view is that we should encrypt
DNS wherever we can and then send it to a new third
party, which then gets access to all your browsing
data. I’m not convinced that that is actually progress.
Previously, we had a highly regulated telecommunica-
tions company, at least here in Europe, that knew what
sites you are visiting. But it’s not progress if now some
American company knows what sites you are visiting.
However, the largest Internet service providers are
now either offering or developing fully encrypted DNS.
So there is not much benefit anymore in sending all
encrypted data to a third party that you did not select.
What one thing should a software engineer remember?
Monitor DNS. When it breaks, everything will break, so
it is worth adding diagnostics. If you have long-lived
applications, the DNS answer that you got at the
start-up of the application that you cached from weeks
ago may no longer be the right IP address. DNS is more
dynamic than it used to be.
GAVIN HENRY is the founder and managing
director of SureVoIP, an Internet telephony
service. Contact him at ghenry@surevoip
.co.uk.

From the analytical engine to the supercomputer,

from Pascal to von Neumann, from punched
cards to CD-ROMs—IEEE Annals of the History
of Computing covers the breadth of computer
history. The quarterly publication
is an active center for the collection and
dissemination of information on historical
projects and organizations, oral history activities,
and international conferences.

www.computer.org/annals

www.computer.org/computingedge 51
 EDITORS: Davide Balzarotti, davide.balzarotti@eurecom.fr
William Enck, whenk@ncsu.edu, Samuel King, kingst@ucdavis.edu
Angelos Stavrou, astavrou@gmu.edu
DEPARTMENT:
SYSTEMS ATTACKS AND DEFENSES
A Cybersecurity Terminarch:
Use It Before We Lose It
Eric Osterweil, George Mason University
W
hy can’t we send encrypted email
(secure, private correspondence that
even our mail providers can’t read)? Why
do our health-care providers require us to use secure
portals to correspond with us instead of directly email-
ing us? Why are messaging apps the only way to send
encrypted messages directly to friends, and why can’t
we send private messages without agreeing to using
a single platform (WhatsApp, Signal, and so on)? Our
cybersecurity tools have not evolved to offer these
services, but why?
Cybersecurity and cryptographically enhanced
tools in the Internet have faced an uphill battle for
many years, due in no small part to the fact that
we do not have a global architecture for deploying
interorganizational (platform-agnostic) verifiability,
authentication, and encryption. This deficiency stems
largely from the absence of a single/unambiguous
global-root cryptographic key for verification [i.e., a
public key that is usable as a global Trust Anchor (TA)].
A global TA could be used to foundationally enhance
protections for security tools, protocols, and more,
but attempts to deploy one in the Internet have a long
history of failure.
To date, there has only been one success story,
and, fortunately, it is still operating. Today, almost
everything we do online begins with a query to a
single-rooted hierarchical global database, whose
namespace is collision-free, and which we have relied
on for more than 30 years: the Domain Name System
(DNS). Moreover, although the DNS protocol did not
initially have verification protections, it does now: the
Digital Object Identifier 10.1109/MSEC.2020.2989703
Date of current version: 9 July 2020
52 July 2021 Published by the IEEE Computer Society 
This article originally
appeared in
vol. 18, no. 4, 2020
term · in · arch
/’ t rm , närk/
noun
an individual that is the last of its species
or subspecies. Once the terminarch dies, the
species becomes extinct.
DNS Security Extensions (DNSSEC). DNSSEC’s pro-
tections stem from the DNS tree’s root TA [often called
the Root Zone’s Key Signing Key (Root KSK)].
Since its deployment, the management, mainte-
nance, and policies surrounding the Root KSK have
been overseen by an international multistakeholder
community, the Internet Corporation for Assigned
Names and Numbers (ICANN) community.1 This model
ensures that there is no single entity that has unilat-
eral jurisdiction over the Root KSK. Today, DNSSEC’s
protocol, policies, and infrastructure are operationally
mature and widely distributed. However, these pro-
tections lie at the low level of the Internet’s founda-
tion and are not often noticed at the application (or
other user-facing) layer(s). This has left the potential
to extend DNSSEC’s verification protections largely
untapped. Moreover, the model we are using exposes
systemic vulnerabilities.
Since the late 1990s, the verification and authen-
tication used by essentially all our security protocols
have made do with a collision-prone hierarchical verifi-
cation model called the Web public-key infrastructure
(Web PKI). Interfaces like secure sockets use the Web
PKI so that applications can benefit from protections
from this model without needing to delve into its com-
plexity. Under these covers, the Web PKI’s verification
2469-7087/21 © 2021 IEEE
 substrate uses multiple roots (i.e., multirooted veri-
fication). It is a loosely organized list of certification
authorities (CAs) that our software uses to verify
essentially all our interorganizational data and trans-
actions (TLS tunnels, HTTPS, secure SMTP, file encryp-
tion, etc.). The Web PKI provides verification and also
asserts trust, but these are separable protections.
Although the Web PKI has raised the bar on mis-
creants, it has also left important security doors
unguarded. Without a definitive starting point for
verification, multirooted verification hierarchies suffer
from architectural vulnerabilities. For instance, when
a multirooted verification hierarchy like the Web PKI
is used to authenticate HTTPS web transactions, rely-
ing party software packages (like web browsers) have
no way to know which CA is authorized to vouch for a
website. This is a problem because it can lead to attes-
tation collisions; whereby browsers have no choice
but to trust any certificate that is verified by any CA.
One of the earliest high-profile exploits of this
attack vector was on a CA named DigiNotar in 2011.2
That event served as a large-scale existence proof
of the vulnerability of this model. In multirooted
hierarchies, RPs generally don’t even have a way to
know who all the roots are supposed to be. Knowing
all the CA roots in the Web PKI is an open challenge.
Browsers attempt to stay current with each other’s
trust stores, and there is an organization called the
CA/Browser (CAB Forum) to coordinate this, but other
systems that have tried to use TLS (HTTPS’s under-
lying secure connection protocol) have found this
intractable. Uses in software like mail servers have
essentially become nonstarters for that reason.
Single-rooted verification hierarchies (like tradi-
tional PKIs) address the aforementioned problems at
an architectural level. With a single root, there is no
ambiguity about which signing authority is allowed to
vouch for whom (the single root clearly disambiguates
this), and there is only one single (well-known) root for
RPs to bootstrap and maintain.
www.computer.org/computingedge
There have been several community attempts
to create single-rooted verification hierarchies for
Internet services, but single-rooted verification has
proven to be a difficult species to breed. In 1989,
there was an attempt to create a single global root for
Privacy-Enhanced Mail in RFC-1114, but the question
of who would operate that root was never answered.
Later the Web PKI began deployment but not as a
multirooted hierarchy. In 1995, VeriSign, Inc. had its CA
certificate configured in the then-dominant Netscape
Navigator web browser.3
At that time, secure web connections verified all
websites’ cryptographic keys by tracing secure delega-
tion paths from that single root. However, the Web PKI
did not have protections that mandated a single root,
and as browser software continued to diversify and the
World Wide Web continued to grow, the list of root CAs
also grew. Today, there are hundreds of root CAs in the
Web PKI that are configured in browsers, and they are
often maintained (i.e., rolled over, revoked, or otherwise
changed) in nontransparent/nonstandard ways.
More recently, the Internet Engineering Task Force
(IETF) has standardized protocols for verifying the
proper holders of Internet Protocol (IP) addresses and
autonomous system numbers using an architecture
called the Resource Public-Key Infrastructure (RPKI) in
RFC-6480. After years of trying to align Internet stake-
holders, and even after unambiguous advice from the
Internet Architecture Board in 2010, the RPKI has not
been able to agree on a single root and now plans to
operate in perpetuity as a multirooted hierarchy. Just
as with the Web PKI, the RPKI now has attestation
collisions. Missed attempts like these underscore the
singular opportunity that DNSSEC represents. It has
even succeeded at an operational scale that other
large (private) single-rooted PKIs have failed. As the
first and only example of a deployed Internet-scale
single-rooted verification hierarchy species, one could
worry that we will not be able to create and operation-
alize another.
53
SYSTEMS ATTACKS AND DEFENSES
New standards, like the DNS-based Authenti-
cation of Named Entities (DANE) suite (RFC-6698,
RFC-8162, and RFC-7929)4 have emerged that have
the immediate potential to be used to unambiguously
secure our protocols and data by using DNSSEC.
At a time when recent global events have caused a
dramatic increase in teleworking, distance learning,
and reliance on online communications and when our
Internet privacy has become top-of-mind to many, why
shouldn’t we be able to apply end-to-end encryption
and object-security to our online lives?
Looking forward, this should also include email,
medical records, cybersecurity information sharing,
and much more. Indeed, now is the right time for us to
reassess the foundations that we have built our pro-
tections on, and also consider if we may be undermin-
ing our strongest (and largely untapped) foundational
component.
PROBLEMS ON THE HORIZON
Recent DNSSEC-based protocols, like DANE, enable
rich protections and are within our grasp; that is, if we
don’t lose them before we use them. Several recent
proposals for DNS over HTTPS, DNS over TLS, and
even DNS over QUIC aim to add security and privacy
protections in ways that would actually create verifi-
cation loops and thereby fundamentally (albeit inad-
vertently) jeopardize the security, stability, and resil-
iency (SSR) of the DNSSEC.
These proposals focus on using transport-layer
security protections, derived from verification per-
formed by the Web PKI, to access DNSSEC. This would
be a disastrous weakness because it would fundamen-
tally undercut DNSSEC’s verification substrate. Our
single-rooted verification hierarchy would (effectively)
become a subtree under the multirooted Web PKI.
DNSSEC’s protections would be subordinated and
thereby have an architectural dependency. Although
the proposal to protect the transport of DNS is well
intended, the approach must not come at the greater
cost of our architectural correctness. If we aren’t con-
scientious in our designs, we may never get another
global single root, making DNSSEC a terminarch (i.e.,
the last of its kind).
Other proposals for alternate (i.e., competing)
DNS roots (such as the Yeti DNS project) threaten
potential disaster, alternate naming schemes (like
54 ComputingEdge
those of Namecoin, Ethereum Name Service, or the
GNU Name System) could portend extinction, and
alternate verification schemes like certificate trans-
parency, which essentially enshrine a global default
trusted source (the way the Web PKI started), herald
the same. In its current state, the DNS has been an
extensible resource for more than 30 years, and we
have only just begun to tap its potential as a cyberse-
curity substrate.
HOW SECURE IS DNSSEC? YOU
BE THE JUDGE!
The operations of the DNSSEC root zone follow the
strictest form of security hygiene. The DNSSEC root
follows a DNSSEC Practice Statement (which is pub-
lished and available to anyone to inspect at iana.org);
its cryptographic keys are maintained in FIPS 140-2
Level 4 hardware security modules, the process and
installations have achieved SOC 3 certification for nine
consecutive years, there are multiple geographically
distributed disaster recovery sites, and high-value
top-level domains (TLDs) (like .com, .net, and so on)
are also following the same level of security practices.
Even routine processes like generating zone
signing keys require a quorum of trusted community
representatives to be physically present and to verify
material, transactions, and adherence to the ICANN
community processes. The DNSSEC root and TLDs are
being meticulously protected and managed to ensure
the SSR of the entire DNS hierarchy with the same
level of security as CAs.
Even exceptions are treated with the highest level
of prudence. In 2017, the Root KSK was scheduled
to gracefully transition to a newer key (i.e., rollover)
because of proactive operational hygiene. However,
before this rollover was executed, measurements indi-
cated a potential problem, which prompted operators
to postpone this rollover. This prudence serves as just
one of many examples of the diligence and care that
exists in the management of DNSSEC’s global root
key. In short, experts are plugged in and taking every
precaution necessary to ensure proper operation and
success of this critical resource.
DEPLOYING DNSSEC
Now that DNSSEC is enjoying broad adoption by
service providers, there are increasingly easy ways
July 2021

for administrators to deploy it. For those who oper-
ate their own DNS infrastructure, almost all mod-
ern DNS name server software platforms can enable
DNSSEC via trivial configurations. For those who use
DNS registrars or other managed DNS (mDNS) pro-
viders to operate their DNS infrastructures, many
of these providers offer to deploy and manage DNS-
SEC through configuration pages in their online por-
tals. Often, simply looking at the existing config-
uration options of either one’s own infrastructure
or the pages of one’s provider can be the one-stop
shopping for turning DNSSEC on. The DNSSEC com-
munities want to help, and resources like the Inter-
net Society’s Deploy 3605 pages offer resources for
guidance to answer questions and to otherwise help
with deploying DNSSEC.
USE IT BEFORE WE LOSE IT
Although using the DNS to resolve a domain name
to an IP address is a critical starting point for almost
all our transactions on the Internet, the DNS was
designed to be used to look up essentially arbitrary
data. Based on this, DNSSEC has evolved the DNS
into a global PKI. DNSSEC is in a position to implement
general-purpose object security. This could be used to
secure threat intelligence, cybersecurity information
sharing, the Internet of Things, email, electronic pro-
tected health information (e-PHI), and much more.
By using DNSSEC as the Internet’s global single
root, we will get interoperability across all those Inter-
net systems that already speak DNS. With DNSSEC,
we can secure caches against poisoning attacks and
reduce transitive trust attack surfaces.6 The DNSSEC
deployment is now counted in the millions of domains
and has been growing and succeeding at exponential
rates, and operators are becoming increasingly adept
at managing their cryptographic deployments.7 It is
ready to be built on.
Protocols like DANE propose to do exactly that,
thereby repairing architectural holes that have
hamstrung HTTPS. This will allow us to enable fallow
object-security models, like using S/MIME for secure
end-to-end email encryption and signing.4 This would
not only plug existing security holes, it would also
give us never-before-seen cybersecurity facilities.
Even DANE’s nascent deployment already numbers
in the hundreds of thousands. It has opened new
www.computer.org/computingedge
SYSTEMS ATTACKS AND DEFENSES
possibilities for Internet cybersecurity companies,
start-ups, and more.
T his all stems from DNSSEC’s global single root,
the DNS Root KSK, the multistakeholder policy
community, and the state-of-the art operations that
support the cryptographic material. Cybersecurity
professionals have the opportunity now to embrace
this resource and capitalize on it to bring the Inter-
net’s cybersecurity to a new high watermark, but
we as a community need to continue to carefully
protect the Internet’s first (and possibly last) global
single-root verification terminarch: DNSSEC.
REFERENCES
1. “ICANN’s multistakeholder model,” ICANN. [Online].
Available: https://www.icann.org/community
2. D. Fisher, “Final report on DigiNotar hack shows total
compromise of CA servers,” Threatpost, Oct. 31, 2012.
[Online]. Available: https://threatpost.com/final
-report-diginotar-hack-shows-total-compromise-ca
-servers-103112/77170/
3. N. Wingfield, “Digital IDs to help secure Internet,”
InfoWorld, Oct. 23, 1995. [Online]. Available: https:
//tinyurl.com/yaf6cwun
4. “How DANE strengthens security for TLS, S/MIME, and other
applications,” Verisign, Nov. 19, 2015. [Online]. Available:
https://blog.verisign.com/security/how-dane-strengthens
-security-for-tls-smime-and-other-applications/
5. Internet Society. Accessed on: May, 9, 2020. [Online].
Available: https://www.internetsociety.org/deploy360
/dnssec/
6. E. Osterweil, D. McPherson, and L. Zhang, “The shape
and size of threats: Defining a networked system’s
attack surface,” in Proc. 2014 IEEE 22nd Int. Conf.
Network Protocols (ICNP), pp. 636–641. doi: 10.1109
/ICNP.2014.101. [Online]. Available: https://ieeexplore
.ieee.org/abstract/document/6980440
7. “Growth and health metrics for the global deployment,”
2020. [Online]. Available: http://secspider.net/
ERIC OSTERWEIL is the vice-chair of the ICANN Second
Security, Stability, and Resiliency Review Team (SSR2 RT).
Osterweil received a Ph.D. from the Computer Science
Department, George Mason University, Fairfax, Virginia.
Contact him at eoster@gmu.edu.
55
IEEE Internet Computing delivers novel content
from academic and industry experts on the
latest developments and key trends in Internet
technologies and applications.

Written by and for both users and developers,

the bimonthly magazine covers a wide range
of topics, including:
• Applications
• Architectures
• Big data analytics
• Cloud and edge computing
• Information management
• Middleware
• Security and privacy
• Standards
• And much more

In addition to peer-reviewed articles,

IEEE Internet Computing features industry
reports, surveys, tutorials, columns, and news.

www.computer.org/internet

VOLUME 22, NUMBER 2 MARCH/APRIL 2018

IEEE INTERNET COMPUTING

VOLUME 22, NUMBER 4 JULY/AUGUST 2018

 July/August 2018
IEEE INTERNET COMPUTING

IEEE INTERNET COMPUTING

Evolution of Rack-Scale Systems

VOLUME 22, NUMBER 1 JANUARY/FEBRUARY 2018

 January/February 2018

VOLUME 22, NUMBER 3 MAY/JUNE 2018

 May/June 2018
IoT-Enhanced Human Experience

Connected and Autonomous Vehicles

IoT-Enhanced Human Experience

Evolution of Rack-Scale Systems
Volume 22
Number 1

Volume 22

www.computer.org/internet
Number 4

Healthcare Informatics and Privacy

www.computer.org/internet

Join the IEEE Computer Society Connected and Autonomous Vehicles

for subscription discounts today!
Volume 22

www.computer.org/internet
Number 3

www.computer.org/internet

www.computer.org/product/magazines/internet-computing
Get Published in the IEEE Open
Journal of the Computer Society

Submit a paper today to the

premier open access journal
in computing and information
technology.

Your research will benefit from

the IEEE marketing launch and
5 million unique monthly users
of the IEEE Xplore® Digital Library.
Plus, this journal is fully open
and compliant with funder
mandates, including Plan S.

Submit your paper today!

Visit www.computer.org/oj to learn more.
 Conference Calendar
I EEE Computer Society conferences are valuable forums for learning on broad and dynamically shifting top-
ics from within the computing profession. With over 200 conferences featuring leading experts and thought
leaders, we have an event that is right for you. Questions? Contact conferences@computer.org.
AUGUST
9 August
• ICKG (IEEE Int ’ l Conf. on
K n o w l e d ge G r a p h), Ho n g
Kong
11 August
• IRI (IEEE Int’l Conf. on Infor-
mation Reuse and Integration
for Data Science), virtual
13 August
• SmartIoT (IEEE Int’l Conf. on
Smar t Internet of Things),
Jeju, South Korea
16 August
• ACSOS (IEEE Int’l Conf. on
Autonomic Computing and
Self- Or ganizing S ys tems),
Washington, DC, USA
18 August
• HOTI (IEEE Symposium on
High-Performance Intercon-
nects), virtual
23 August
• AITest (IEEE Int’l Conf. on Arti-
ficial Intelligence Testing),
virtual
• BigDataSer vice (IEEE Int’ l
Conf. on Big Data Comput-
ing Service and Applications),
virtual
• COINS (IEEE Int’l Conf. on
Omni-Layer Intelligent Sys-
tems), Barcelona, Spain
• DAPPS (IEEE Int’l Conf. on
58 July 2021
Decentralized Applications
and Infrastructures), virtual
• JCC (IEEE Int’l Conf. on Joint
Cloud Computing), virtual
• MobileCloud (IEEE Int’l Conf.
on Mobile Cloud Computing,
Services, and Engineering),
virtual
• SCC (IEEE Space Computing
Conf.), virtual
• SMARTCOMP (IEEE Int’l Conf.
on Smart Computing), Irvine,
USA
• SOSE (IEEE Int’l Conf. on Ser-
vice-Oriented System Eng.),
virtual
25 August
• TASE (Int’l Symposium on The-
oretical Aspects of Software
Eng.), Shanghai, China
SEPTEMBER
1 September
• CBI (IEEE Int’l Conf. on Busi-
ness Informatics), virtual
5 September
• SERVICES (IEEE World Con-
gress on Services), Chicago,
USA
7 September
• ASYNC (IEEE Int’l Symposium
on Asynchronous Circuits
and Systems), virtual
• CLUSTER (IEEE Int’l Conf. on
Published by the IEEE Computer Society
Cluster Computing), Portland,
Oregon, USA
• EuroS&P (IEEE European Sym-
posium on Security and Pri-
vacy), Vienna, Austria
8 September
• MIPR (IEEE Int’l Conf. on Mul-
timedia Information Process-
ing and Retrieval), Tok yo,
Japan
10 September
• E W D T S ( I E E E E a s t- W e s t
Design & Test Symposium),
Batumi, Georgia
13 September
• BCD (IEEE/ACIS Int’l Conf. on
Big Data, Cloud Computing,
and Data Science), Zhuhai,
China
20 September
• AI4I (Int’l Conf. on Artificial
Intelligence for Industries),
Laguna Hills, USA
• eScience (IEEE Int’l Conf. on
eScience), Innsbruck, Austria
• RE (IEEE Int’l Requirements
Eng. Conf.), virtual
• TransAI (Int’l Conf. on Trans-
disciplinary AI), Laguna Hills,
USA
27 September
• ICSME (IEEE Int’l Conf. on
Sof tware Maintenance and
Evolution), virtual
2469-7087/21 © 2021 IEEE
OCTOBER Workshop), Washington, D.C., on Workload Characteriza-
1 October USA tion), virtual
• ISPA (IEEE Int’l Symposium on 13 October 15 November
Parallel and Distributed Pro- • FIE (IEEE Frontiers in Educa- • ASE (IEEE/ACM Int’l Conf. on
cessing with Applications), tion Conf.), Lincoln, Nebraska, Automated Sof tware Eng.),
New York, USA USA Melbourne, Australia
4 October • WF-5G (IEEE 5G World Forum), • BigMM (IEEE Int’l Conf. on Mul-
• IC2E (IEEE Int’l Conf. on Cloud Montreal, Canada timedia Big Data), Taichung,
Eng.), San Francisco, USA 16 October Taiwan
• ISMAR (IEEE Int’l Symposium • MICRO (IEEE/ACM Int’l Sym-
on Mixed and Augmented posium on Microarchitecture), DECEMBER
Reality), Bari, Italy Athens, Greece 20 December
• LCN (IEEE Conf. on Local Com- 17 October • MCSoC (IEEE Int’l Sympo -
puter Networks), Edmonton, • ICVRV (IEEE Int’l Conf. on Vir- sium on Embedded Multicore/
Canada tual Reality and Visualization), Many-Core Systems-on-Chip),
• MASS (IEEE Int’ l Conf. on Nanchang, China Singapore
Mobile Ad Hoc and Smart Sys- 18 October
tems), Denver, USA • SecDev (IEEE Secure Develop-
6 October ment Conf.), Atlanta, USA
• DFT (IEEE Int’l Symposium on 21 October
Defect and Fault Tolerance in • IEEE Cloud Summit, Hemp-
VLSI and Nanotechnology Sys- stead, New York, USA
tems), virtual 24 October
10 October • VIS (IEEE Visualization Conf.),
• M O DEL S (AC M/IEEE In t ’ l New Orleans, USA
Conf. on Model Driven Eng. 25 October
Languages and S ystems), • EDOC (IEEE Int’l Enterprise
Fukuoka, Japan Distributed Object Computing
11 October Conf.), Gold Coast, Australia
• ESEM (ACM/IEEE Int’l Sympo-
Learn more
sium on Empirical Software NOVEMBER
Eng. and Measurement), Bari, 2 November about IEEE
Italy • ICNP (IEEE Int’l Conf. on Net- Computer
• ICCV (IEEE/CVF Int’l Conf. on
Computer Vision), Montreal,
work Protocols), Dallas, USA
6 November
Society
Canada • SmartCloud (IEEE Int’l Conf. on conferences
12 October Smart Cloud), Newark, USA
computer.org/conferences
• AIPR (IEEE Applied Imag- 7 November
e r y Pa t t e r n R e c o g n i t i o n • IISWC (IEEE Int’l Symposium
IEEE COMPUTER SOCIETY ELECTION

Volunteer Leadership

Is Vital
Vote by Monday, 20 September at 12PM EDT

www.computer.org/election2021