Dark Data: Are We Solving the Right Problems?
Michael Cafarella1, Ihab F. Ilyas2, Marcel Kornacker3, Tim Kraska4 and Christopher Ré5
1
University of Michigan, USA
2
University of Waterloo, USA
3
Cloudera, USA
4
Brown University, USA
5
Stanford University, USA
michjc@umich.edu, ilyas@uwaterloo.ca, marcel@cloudera.com, tim kraska@brown.edu, chrismre@cs.stanford.edu
Abstract—With the increasing urge of the enterprises to ingest
as much data as they can in what’s commonly referred to as
“Data Lakes”, the new environment presents serious challenges
to traditional ETL models and to building analytic layers on top
of well-understood global schema. With the recent development
of multiple technologies to support this “load-first” paradigm,
even traditional enterprises have fairly large HDFS-based data
lakes now. They have even had them long enough that their first
generation IT projects delivered on some, but not all, of the
promise of integrating their enterprise’s data assets. In short, we
moved from no data to Dark data. Dark data is what enterprises
might have in their possession, without the ability to access it or
with limited awareness of what this data represents. In particular,
business-critical information might still remain out of reach.
This panel is about Dark Data and whether we have been
focusing on the right data management challenges in dealing
with it.
I. W HY IS THE TOPIC OF CURRENT INTEREST TO THE
ICDE COMMUNITY ?
Big data and the ability to deal with all of its aspects (e.g.,
volume, variety, etc.) is a core activity to the data management
community. It is easy to see that with the availability and the
affordability of Big Data technologies, the huge amount of
data enterprises have been collecting has caught the research
community off-guard. In response, academia has been largely
focused on revising the known techniques and algorithms to
work on a large scale in a scale-out or a scale-up environment.
On the other hand, there has been an explosion in the number
of start-ups in the big data space with more pragmatic solutions
to solve the immediate needs of the enterprise, including
data ingestion, elasticity, provenance, accelerating analytics,
engaging data scientists, monitoring operations, and helping
IT personnel to manage the continually growing data asset and
the technology stacks they are deploying. We strongly believe
that we need to re-evaluate where we have been spending our
effort in lieu of the real problems “big data” owners have.
II. W HY IS THE TOPIC WELL - SUITED TO AN OPEN PANEL
DISCUSSION ?
Almost all large enterprises we talked to confirmed that
there is a huge confusion in the market on the right tech-
nologies, architectures, standards, etc. We strongly believe
that there is a similar amount of confusion in the research
community on the right place to invest. An open discussion is a
great medium to surface this confusion and to help understand
the spectrum of activities involved in this space. In particular,
consider the following issues:
978-1-5090-2020-1/16/$31.00 © 2016 IEEE 1444
• Schema on Read: One of the big promises of Big Data
solutions is the load-first paradigm. What is the impact
of this promise on algorithms, and the assumptions we
make in our settings.
• Schema extraction and evolution: with data streaming
in from a variety of sources, manual curation of a
global schema becomes impossible. Instead, we need
to look for ways to automate the process of schema
extraction and schema evolution.
• Information Extraction Processes: Information extrac-
tion has been a research interest since at least the
early 1990s, but will likely reach new importance as a
central element of any effort that combines dark data
with traditional relational tools. What is the best way
to design an extraction system that transforms dark
data into relational data?
• Continuous Data Curation: For decades, we have been
treating ETL and cleaning as a one-shot exercise
thet is tied closely to the ingest box in business
intelligence stacks. Well, load-first paradigms and the
swamps of dark data calls for a different approach
to the curation process as a continuous, incremental,
fuzzy, and interactive process. What does that mean
to the large body of work the database community
contributed in this domain.
• Reasoning about Data Quality: Datasets in the data
lakes are both incredibly numerous and exhibit incred-
ible variation in quality. Fully cleaning the data lake
will likely never be possible. How can we build data
applications that are resilient to low-quality inputs,
or at least alert the user when inputs are not clean
enough?
• Machine Learning?: Machine learning may make it
radically easier to attack both classical applications
and open new dark data applications. But machine
learning is not a panacea. It still requires a huge effort
to deploy, tune, and run. Does the data management
community play a role in transitioning machine learn-
ing from “demoware” to an every day data processing
system? Does it change how we ask and answer
questions?
Moreover, machine learning researchers have histor-
ically not devised abstractions that are critically im-
portant to building a long lasting system. The data
management community has. What does the future of
these systems look like with data constantly in motion,
ICDE 2016 Conference
 and when new data is continuously arriving and being
joined to historical data?
• Practical Provenance: The full set of processing steps
applied to a specific data lake—including extraction,
cleaning, schema matching operations, and traditional
relational algebra steps—is likely to be extremely
complex and in many cases the result of ad-hoc pro-
grams. But government auditors and machine learning
safety guarantees require we know the full prove-
nance of certain files. How can we retain sufficient
provenance information while allowing users extreme
flexibility in performing these steps?
III. W HAT SHOULD ATTENDEES EXPECT TO GAIN FROM
THE PANEL ?
We hope that the panel discusses some of the controversies
around dealing with Dark Data in Big Data environments. At
the least, we will highlight multiple new realistic and pragmatic
challenges that the community needs to solve or tackle to have
a leading and impacting role.
IV. PANELLISTS
Michael Cafarella
Michael Cafarella is an assistant professor in the division
of Computer Science and Engineering at the University of
Michigan. His research interests include databases, information
extraction, data integration, and data mining. With Doug
Cutting he co-founded the Nutch and Hadoop open source
projects. He is also a co-founder of Lattice Data, a startup that
extracts relational databases from dark data.
Ihab F. Ilyas
Ihab Ilyas is a professor in the Cheriton School of Com-
puter Science at the University of Waterloo. He received
his PhD in computer science from Purdue University, West
Lafayette. His main research is in the area of database systems,
with special interest in data quality, managing uncertain data,
rank-aware query processing, and information extraction. Ihab
is a recipient of the Ontario Early Researcher Award (2009), a
Cheriton Faculty Fellowship (2013), an NSERC Discovery Ac-
celerator Award (2014), and a Google Faculty Award (2014),
and he is an ACM Distinguished Scientist. Ihab is a co-founder
of Tamr, a startup focusing on large-scale data integration and
cleaning. He serves on the VLDB Board of Trustees, and he
is an associate editor of the ACM Transactions of Database
Systems (TODS).
Marcel Kornacker
Marcel Kornacker is the Chief Architect for database
technology at Cloudera and founder of the Cloudera Impala
project. Following his graduation in 2000 with a PhD in
databases from UC Berkeley, he held engineering positions
at several database-related start-up companies. Marcel joined
Google in 2003 where he worked on several ads serving and
storage infrastructure projects, then became tech lead for the
distributed query engine component of Google’s F1 project.
1445
Tim Kraska- (Moderator)
Tim Kraska is an assistant professor in the Computer
Science department at Brown University. Currently, his re-
search focuses on Big Data management systems for modern
hardware and new types of workloads, especially machine
learning. Before joining Brown, Tim Kraska spent 2 years
as a PostDoc in the AMPLab at UC Berkeley after receiving
his PhD from ETH Zurich, where he worked on cloud-scale
data management systems and stream processing. He was
awarded an NSF Career Award (2015), an AFOSR Young In-
vestigator award (2015), a Swiss National Science Foundation
Prospective Researcher Fellowship (2010), a DAAD Scholar-
ship (2006), a University of Sydney Master of Information
Technology Scholarship for outstanding achievement (2005),
the University of Sydney Siemens Prize (2005), two VLDB
best demo awards (2015 and 2011) and an ICDE best paper
award (2013).
Christopher Ré
Christopher (Chris) Ré is an assistant professor in the
Department of Computer Science at Stanford University. His
work’s goal is to enable users and developers to build ap-
plications that more deeply understand and exploit data. Chris
received his PhD from the University of Washington in Seattle
under the supervision of Dan Suciu. For his PhD work in prob-
abilistic data management, Chris received the SIGMOD 2010
Jim Gray Dissertation Award. He then spent four wonderful
years on the faculty of the University of Wisconsin, Madison,
before moving to Stanford in 2013. He helped discover the
first join algorithm with worst-case optimal running time,
which won the best paper at PODS 2012. He also helped
develop a framework for feature engineering that won the
best paper at SIGMOD 2014. In addition, work from his
group has been incorporated into scientific efforts including
the IceCube neutrino detector and PaleoDeepDive, and into
Cloudera’s Impala and products from Oracle, Pivotal, and
Microsoft’s Adam. He received an NSF CAREER Award in
2011, an Alfred P. Sloan Fellowship in 2013, a Moore Data
Driven Investigator Award in 2014, the VLDB early Career
Award in 2015, and the MacArthur Foundation Fellowship in
2015.