2015 48th Hawaii International Conference on System Sciences
Data Ships:
An Empirical Examination of Open (Closed) Government Data
Karine Nahon
University of Washington
The Interdisciplinary Center (IDC) Herzliya
karineb@uw.edu
Abstract
As part of endorsing the open government data
movement in many parts of the world, governments
have worked to increase openness in actions where
information technologies play a major role.
Releasing public data was perceived by many
governments and officials as a fundamental element
to achieve transparency and accountability. Many
studies have criticized this approach and illustrated
that open government data does not necessarily lead
to open government. Our study examines for the first
time in a systematic, quantitative way the status of
open government data in the US, by focusing on the
disclosure of data by US federal agencies. Our
findings suggest that most US federal agencies
largely do not follow the open government policies of
2009 and 2013. The paper discusses the type of
public data that is released, and analyzes the
(non)strategy of its release.
1. Introduction
Open government is a concept coined decades
ago. Yu and Robinson credit Wallace Parks, who
served as counsel to the Special Subcommittee on
Government Information, as the first to expound on
the term in print in his 1957 article “The Open
Government Principle: Applying the Right to Know
Under the Constitution” [1]. It was not, however,
until 2009, when US President Barack Obama issued
the Open Government Directive (OGD) 1.0 [2], and
later established the Open Government Partnership,
that this concept became mainstream. Since then,
many democratic countries have joined the global-
and-local open government movement. This
movement has resulted in diverse and dispersed
initiatives, metrics, methodologies, and data release
practices. The translation of the goals of the open
government movement into practical programs faced
many obstacles. For example, Yu and Robinson
analyzed the vagueness of the concepts “open
government” and “open data,” which present multiple
meanings to policy makers, civil society, academia,
and practitioners [1]. Peled illustrated inherent
1530-1605/15 $31.00 © 2015 IEEE
DOI 10.1109/HICSS.2015.264
Authorized licensed use limited to: North Carolina Central University. Downloaded on November 27,2021 at 05:17:46 UTC from IEEE Xplore. Restrictions apply.
Alon Peled
The Hebrew University of Jerusalem
alon.peled@post.harvard.edu
contradictions in the primary goals of open
government, showing that transparency and
collaboration are potentially competing forces [3].
Scholarship in the first years of the open
government movement was characterized by utopian
voices, enthusiastically emphasizing the potential and
actual benefits of open data as a fundamental element
to achieve transparency and accountability; and a
mechanism for innovation, improved efficiency,
participation and self-empowerment. The architects
of open government programs argued that openness
mobilized the expertise of the masses, harnessed
agencies’ hidden desire to share data, and gave birth
to a community of innovators. In open government
jargon, individuals were “called to arms” to support
the open government program; everything associated
with “the government of the past” had to be brushed
away, and the spirit of openness and innovation was
claimed to be contagious [4], [5],[6].
These voices were later replaced with studies
assessing the benefits and barriers of open
government initiatives [7], [8] and proposing ways to
measure the progress of these initiatives [9].
However, most of these studies relied on case studies
or limited empirical analysis (qualitative and
quantitative).
In the US, as a response to the OGD [2] federal
agencies developed plans to increase openness and
transparency in actions where information
technologies play a major role [1],[9],[10],[11].
Government transparency occurs through four
primary channels: proactive dissemination by the
government; release of requested materials by the
government; public meetings; and leaks from
whistleblowers [12]. Therefore, the release of public
data was perceived by US government officials as a
fundamental method to achieve transparency and
accountability.
In reality, there was a big gap between open
government data programs and the goals of the open
government movement. This gap was evidenced by
poor data quality; release of data irrelevant to
government service; outdated, broken links; and
2209
 restrictions on data use [1]. Open Government
proponents claimed that outcomes of the open
government movement can be achieved, but data
must first be available, accessible and usable.
By developing an automated system to crawl into
open government data portals of US federal agencies,
we can for the first time assess in a systematic way
the real status of open federal data in the US.
Specifically, this paper discusses the following
questions: Did the open government data disclosure
in the US achieve the goals of its declared open
government policy? Were agencies compliant with
open data policy? What are the barriers to and
opportunities for making government data open in the
US?
2. Open Government Data in the US
The US federal government was defined as the
largest producer, collector, consumer, curator and
disseminator of information in the US [13],[15].
Hence, President Obama’s 2009 promise to free non-
classified and non-sensitive information raised hopes
that the new avalanche of open, free and easily
accessible government information would unleash
innovation and transparency in the US.
Indeed, immediately after his election, President
Obama unleashed a lightning campaign to enforce
agency openness. On his first full day in office
(January 21, 2009), at the height of the worst
economic crisis America had experienced since the
Great Depression, President Obama signed three
memoranda and two executive orders. Four of these
five documents promoted open government [3]. In
the same month, President Obama reversed the
Ashcroft memorandum of October 2001, which
advised agencies to employ caution in cooperating
with the Freedom of Information Act (FOIA). Obama
declared: “In the face of doubt, openness prevails”
[16]. In March 2009, Vivek Kundra was appointed as
the first federal Chief Information Officer (CIO),
signaling the importance of the program to the US
government. A barrage of open government
initiatives surfaced, including eRulemaking, IT
Dashboard, Recovery.gov and USAspending.gov.
The administration showcased open government
stories and ensured that senior appointees adhered to
open government principles [20].
On May 21, 2009, a team headed by the CIOs of
the US Department of the Interior (DOI) and the US
Environmental Protection Agency (EPA) launched
www.data.gov as the premier web publishing
location of the most important federal datasets [2].
Then, on December 8, 2009, the Office of
Management and Budget (OMB) published the OGD
2210
Authorized licensed use limited to: North Carolina Central University. Downloaded on November 27,2021 at 05:17:46 UTC from IEEE Xplore. Restrictions apply.
[2]. The directive ordered agencies to make as much
information as possible available online. At the
outset, agencies were instructed to publish at least
three “high-value” datasets on the web. High-value
datasets were defined as containing “information that
can be used to increase agency accountability and
responsiveness; improve public knowledge of the
agency and its operations; further the core mission of
the agency; create economic opportunity; or respond
to need and demand as identified through public
consultation” [2].
High-value datasets were required to have never
been made available or published in a downloadable
and open format [2]. The first three high-value
datasets were considered a down payment, or a
minimum; agencies were expected to continually
make new datasets available to the public [17]. The
OGD specifically demanded that agencies “identify
additional high value information not yet available
and establish a reasonable timeline for publication
online in open formats with specific target dates” [2],
[7]. However, at the 2009 OGD, only federal
departments and non-independent federal agencies
were instructed to comply with the OGD.
Four years later, in May 2013, on commencing his
second term, President Obama raised the open
government bar with a memorandum ordering that
the default state of new and updated government
information resources be made open and machine-
readable. He also instructed that government
information be managed as an asset throughout its
life cycle. Agencies were required to maintain and
nurture their datasets as products with real market
value; that is, as assets that could be bought, sold and
exchanged [18]. In this article, we adopt the concept
of an “asset” and refer to the information artifacts
that agencies generate, maintain and upgrade as
information assets.
To ensure these new orders would be followed,
President Obama again enforced a rigorous open
government data execution timetable with which
agencies were required to comply. He also
established a Cross-Agency Priority Goal to track the
implementation of the second-generation of the open
government policy. Finally, President Obama
extended the OGD policy to all federal agencies by
instructing independent agencies to comply with the
policy [19].
Following the 2013 memorandum, OMB
published a detailed implementation guide instructing
agencies how to follow the President’s latest OGD
policy. Henceforth, we refer to this implementation
guide as the “US federal open data 2.0 standard.”
First and most importantly, all federal agencies were
ordered to develop a Public Data Listing, which
 contains a list of all information assets that are or
could be made available to the public. This Public
Data Listing was to be posted at
www.[agency].gov/data.json. The intention was to
provide the public with access to an extensive
inventory of agencies’ information assets, enable the
public to track agencies’ subsequent progress to clear
more assets for publication, and empower the public
to re-use open data. The listing was not designed to
hold metadata about an agency’s classified
information assets such as those concerning national
security or containing sensitive personal information
about citizens.
According to the US federal open data 2.0
standard, agencies were instructed to: continuously
add new information assets (on top of the information
assets that were published on data.gov before August
1, 2013); improve the quality of previously released
information assets; enrich the metadata descriptions
of published information assets so the public could
discover them more easily; and increase the amount
of information shared with other agencies,
corporations, organizations and citizens.
The OMB’s intention was to dynamically
populate the newly renovated data.gov and use the it
as the primary website to find information assets
generated and held by the US government [19]. The
designers of the revised open government data policy
were keenly aware of the problems of publishing and
discovering datasets in the first iteration of the US
federal open data portal (2009-2013) [3]. Beginning
in May 2013, these designers planned to transfer
directly to federal agencies the full responsibility of
releasing and accurately describing their information
assets on agency-based servers. The new federal open
data portal (currently in Beta) was designed to be a
secondary index of the information assets that federal
agencies publish and describe on their own portals. It
is important to note that, in practice, the original
www.data.gov was never more than a secondary
index of the information assets that federal agencies
published on their own servers (see further
explanation in the method section).
Table 1 summarizes three key differences
between the original OGD 1.0 policy as presented in
the 2009 [2] and the latest version of this policy,
OGD 2.0, as defined by the OMB implementation
guide of 2013 [19]. Our work is the first attempt to
measure — quantitatively and systematically —
whether and how US federal agencies have complied
with President Obama’s open government policy of
2009 and 2013.
2211
Authorized licensed use limited to: North Carolina Central University. Downloaded on November 27,2021 at 05:17:46 UTC from IEEE Xplore. Restrictions apply.
Table 1: Differences between OGD 1.0 and
2.0 policies
OGD 1.0 OGD 2.0
(2009) [2] (2013)
[18], [19]
Instruction for independent Optional Mandatory
agencies to comply with OGD
Publication of comprehensive No Mandatory
data inventory lists
Default creation of all new No Mandatory
information assets as OGD assets
3. Benefits/Barriers of Open Government
Data Programs: It Depends Who You
Ask
Various scholars carefully examined open
government data programs in diverse countries
attempting to understand the benefits and barriers of
these programs [7],[11],[20]. Janssen et al [8]
clustered the benefits of open government data
programs into three categories: 1. Political and social
benefits, such as increasing transparency and
participation, fostering accountability, empowering
citizens, and improving government services; 2.
Economic benefits, such as stimulating innovation,
growing the economy, and developing new products
and services; and 3. Operational and technical
benefits, such as optimizing administration processes,
reusing data, and improving public policies.
In 2011, initial enthusiasm for the global-and-
local open government movement began to fade. A
new body of scholarship emerged claiming that open
government data programs did not deliver intended
outcomes due to various political, organizational,
technological and financial challenges. These
researchers studied the barriers of such programs in
an attempt to fix major flaws in their design and
implementation. For example, Zuiderwijk et al [21]
examined 118 impediments of open government
programs and identified 10 categories of
impediments from the user perspective: 1)
availability and access, 2) find-ability, 3) usability, 4)
understand-ability, 5) quality, 6) linking and
combining data, 7) comparability and compatibility,
8) metadata, 9) interaction with the data provider, and
10) opening and uploading. Peled focused on the
supply side and divided criticism of open government
data programs worldwide into three categories: bad
initial program design, flawed program execution and
adverse program consequences [22]. In this paper we
will focus on the supply side as well.
In the US, the OGD 1.0 gave agencies discretion
to decide what data to publish and to evaluate their
own performance; this allowed agencies to passively
 resist the open data program. Many agencies did not
set openness deadlines or publish performance data;
others refused to share data release plans or did not
live up to the goals that they themselves created. Not
surprisingly, most agencies that assessed their own
performance awarded themselves the highest
compliance ranking [22],[23]. However, it is
important to note that there is not one agreed-upon,
uniform or binding way to measure the “compliance”
of agencies with open government data policies.
These policies themselves were vague from day one,
so it is unfair to rank and grade agencies on
“compliance” with open government data policies.
Still, we believe it is important, even critical, to study
the supply side of open government data programs as
part of an effort to develop smarter and more
effective open government policies and ways to
measure the progress of these policies in the future.
Most agencies reluctantly joined the US open data
program. In mid-2011, 172 American agencies
participated in the program, yet three agencies
uploaded about 99% of the content, and most of it old
1 datasets were previously available. The data lacked
content. The average participating agency had not
returned to www.data.gov for 222 days since its last
data.gov transaction [3]. European agencies, too,
reluctantly participated in open data programs and
dumped volumes of purposeless raw data into
cyberspace [24],[25]. In Britain, Estonia and
Denmark, certain agencies refused to free data
because their income was partially dependent on data
sales[24],[25]. Some scholars suggested that agencies
refused to free datasets because these datasets are
“bargaining chips” in inter-agency relationships.
Other scholars argued that agencies manipulate their
closely held datasets to convince legislatures to grant
them budgets. Finally, scholars argued that because
open data legislation compelled agencies to employ
external consultants, senior government information
technology (IT) officials might have been reluctant to
delegate to consultants the politically sensitive job of
deciding which datasets to release to the public,
choosing instead not to release datasets.
So, scholars hypothesized that the open data
program offered agencies a bad deal: Politicians
received public approval for “freeing data” while
agencies were expected to free valuable datasets and
undertake the time and effort-consuming job of
preparing them for release. Agencies therefore
minimized their open data involvement [3],[25],[26].
1
The three agencies — CENSUS, USGS and NOAA — published
catalogs of downloadable information assets on their agency-based
web portals long before President Obama declared his Open Data
policy. Beginning in 2009, www.data.gov served as a secondary
index to publish the contents of these older catalogs.
2212
Authorized licensed use limited to: North Carolina Central University. Downloaded on November 27,2021 at 05:17:46 UTC from IEEE Xplore. Restrictions apply.
Agencies were also concerned about
decontextualizing their data. Data wrapped in context
and traceable to its sources is a record. Records are
the blood cells of governmental work. Noveck wrote:
“the right of transparency is eviscerated by the
practical inability of all but a handful of professionals
to make sense of information” [4]. But the open
government data program divorced datasets from
their source records, thus converting useful records in
many cases into useless datasets [27],[28]. For
example, the Environmental Protection Agency
(EPA) maintained context-rich Toxic Release
Inventory (TRI) records on its website, which were
sliced and diced into numerous, context-free datasets
and uploaded to www.data.gov in 2010. In addition,
the initial OGD did not prioritize what data to release
first [29] and did not establish mechanisms for
citizens to verify data’s accuracy, completeness and
authenticity [30],[31]. Practitioners argued that
agencies released voluminous and meaningless
datasets, repackaged data goods previously published
elsewhere, and did not indicate whether released
descriptions and, sometimes, datasets could not be
downloaded or opened. Agencies did not offer
mechanisms to report data problems or provide
explanations for the removal of released datasets
[27].
Finally, open government data architects did not
consider the cost of “freeing” data. Kundra argued
that the government spends billions of dollars on
“armies of consultants, a fragmented infrastructure,
and customized, one-off applications” and that open
data citizen-developers “can do more for less” [32].
Yet agencies had to hire staff to understand new
legislation, adjust data to new standards, train
employees, and improve data quality. Agencies also
needed to convert hand-written and verbal data into
digital records and integrate non-compatible data
streams to prepare data for release. These activities
were costly and not included in agencies’ budgets
[30], [33]–[35]. A recent study based on interviews
with 155 senior US federal IT officials revealed that
these officials are concerned about the cost of
adjusting their records management programs to the
demands of the open data program [36]. Nonetheless,
to date, the above claims have not been supported by
a rigorous, empirical-systemic study.
4. Method: Systematically Harvesting and
Analyzing Information Directly from
Federal Agencies
In this section, we compare our novel machine-
based open government data research technique to
 the two other more established research techniques
for analyzing metadata (i.e., agency- and citizen-
based techniques). We then discuss briefly the unique
challenges we confronted while developing our
machine-based research technique.
4.1. Agency-, Citizen- and Machine-Based
Open Government Data Research Techniques
Research techniques which use metadata to assess
the status of open government data programs can be
broadly categorized into three categories: agency-
based, citizen-based and machine-based techniques.
There are other techniques for assessing open
government data programs that do not involve
studying metadata such as historical, document and
policy analysis and more. In this paper we will focus
on techniques, which mainly use metadata analysis.
In analyzing metadata researchers attempt to address
the following five key questions:
1. Impartiality: Is the collected metadata (about the
released information assets) impartial or,
alternatively, biased to reflect the interests of
some parties?
2. Quality: Is the collected metadata of good quality
or, alternatively, is it poorly and hastily collected
and organized?
3. Freshness: Is the collected metadata timely or,
alternatively, was the metadata collected a long
time ago and no longer reflects the evolution of
the information asset that the feedback
information is pretending to measure?
4. Comprehensiveness: Does the collected metadata
include many types of information about the
information assets?
5. Linkability: Can we associate metadata about a
given information asset with the metadata about
another asset?
In the “Agency-Based” technique, agencies are
asked to rank themselves on various scales of
‘compliance’ with open government data policy.
“Agency-Based” is the least reliable of the open
government data measurement research techniques.
Agencies have no interest in reporting partial findings
about their own compliance; nor do agencies possess
incentives to provide high-quality, timely,
comprehensive or linkable metadata about the
information assets they release. So, metaphorically,
the agency-based “measurement technique” is akin to
empowering a cat to watch over the cream. During
his first presidency, President Obama and his OGD
officers relied on this technique and, unsurprisingly,
all agencies awarded themselves the highest rankings
for compliance with the new OGD policy [23].
2213
Authorized licensed use limited to: North Carolina Central University. Downloaded on November 27,2021 at 05:17:46 UTC from IEEE Xplore. Restrictions apply.
The “Citizen-Based” research technique relies on
citizens reporting about their use patterns and
experience with open government data portals.
Citizens often provide high-quality feedback about
metadata of information assets, which they found
useful, including metadata about how they managed
to link different assets. However, citizens are
required to fill out tedious forms about their
experience working with OGD information assets.
Therefore, most information assets that agencies
release are not included in such citizen-based OGD
reports (weak comprehensiveness). In addition,
citizens’ reports are usually a one-time endeavor, and
not a repetitive inquiry, which re-assess periodically
the state of open government data (i.e., poor
freshness). The European Engage program is a good
example of the benefits and challenges of citizen-
based techniques [37]
Finally, in this paper we are pioneering a third
type of evaluation research technique—Machine-
Based. The software2 automatically detects, visits and
re-visits information assets that agencies published
on open government data portals. Unlike humans
(i.e., agency officials or citizens) the software is
impartial, harvests hundreds of metadata fields about
each information asset and is timely (i.e., visiting
information assets in OGD portals frequently to see
whether something new can be learned about this or
that information asset). Our Machine-Based research
technique also provides higher-quality feedback
information than the Agency-Based and Citizen-
Based research techniques because it harvests
additional metadata about the information asset from
the published data itself.
Table 2: A Comparison of the
Agency-, Citizen- and Machine-Based
OGD Research Techniques
Agency- Citizen- Machine-
Based Based Based
Impartiality •- •
Quality • •
Freshness •
Comprehensiveness •
Linkability •
Table 2 summarizes the differences among the
three techniques for analyzing metadata. The main
weakness of metadata analysis is that it does not
study the content of the information assets, and
instead it focuses on the information about the
information assets. Interestingly, in its current state,
our Machine-Based software has one major drawback
in comparison to the Citizen-Based research
technique — it does not provide measurements about
2
Alon Peled, a co-author of this paper, developed the software.
 the potential to link one information asset to another
in the way that citizens do when they report what
they did with data. Linked Open Data is the current
battle cry of the open government data movement, so
we are well aware how important it is to address this
weakness.
4.2. Exploring the Machine-Based Research
Technique
Numerous studies focused on the US federal
www.data.gov portal to analyze the open government
data program in the US. Scholars found it easier to
harvest metadata from a single web portal
(www.data.gov) than to collect similar data from
hundreds of federal open government data sites [38].
However, this trend of placing the data.gov portal at
the center of scholarly research poses critical
challenges to researchers. .
The 2009 OGD [18] explicitly declared that the
core of President Obama’s new open government
program would reside within the open government
3 to systematically gather the information assets and
data portals of “individual federal agencies” :
“Within 60 days, each agency shall create an Open
Government Webpage located at
http://www.[agency].gov/open to serve as the
gateway for agency activities related to the Open
Government Directive and shall maintain and update
that webpage in a timely fashion” [2]. Likewise, the
2013 Presidential memorandum extended this logic,
demanding that “the default state of new and
modernized government information resources shall
be open and machine readable” and accessible on
individual agencies’ open government data portals
[39].
Right from its widely publicized launch day in
2009, the US federal open data portal was never
intended to be more than an index to help end users
discover and access information assets on individual
agency web portals. The US federal open data 2.0
standard instructs each agency to develop its own
Public Data Listing that includes all data assets that
are or could be made available to the public. Each
agency was instructed to post this listing on its own
agency server using the data.json catalog (posted at
4 columnary data into additional metadata descriptions
www.[agency].gov/data.json). The standard explains
3
The term “agency” in this paper refers to either an agency (e.g.,
EPA) or department (e.g., Department of Defense).
4
JSON (JavaScript Object Notation) is a lightweight data-
interchange format. It is easy for humans to read and write. It is
easy for machines to parse and generate. It is based on a subset of
the JavaScript Programming Language, Standard ECMA-262 3rd
Edition - December 1999. JSON is a text format that is completely
language-independent but uses conventions that are familiar to
programmers of various languages. These properties make JSON
an ideal data-interchange language.
2214
Authorized licensed use limited to: North Carolina Central University. Downloaded on November 27,2021 at 05:17:46 UTC from IEEE Xplore. Restrictions apply.
that the newly renovated data.gov federal open
government data portal (currently in its Beta version)
would merely populate its indices by harvesting and
aggregating information from these agency-managed
Public Data Listings [19]. Simply put, the core in all
matters regarding the US federal open government
program was (and still is) to take place on individual
agency web portals, while www.data.gov was
designed to serve as a secondary index of these
information assets. The first Obama Administration
highlighted www.data.gov as the flagship of the open
government program. In reality, www.data.gov was
never more than a glorified, and not especially useful,
secondary index.
Accordingly, we argue that scholars who are
interested in analyzing the promise, challenges and
opportunities of the US federal open government
program must pay closer and more detailed attention
to agencies’ individual open government data portals
rather than to secondary indices such as
www.data.gov. Therefore, our software was designed
their metadata directly from US federal agencies
open government portals, if existed. Most
importantly for this paper, this software solution
crawls, scrapes, collects, cleanses, and registers
existent metadata of information assets harvested
5
from a list of 473 US federal agencies , focusing on
agency open government portals.
The software works in the following way: The
software crawls into an open government data portal
of an agency and performs an initial indexing of all
the information assets published there by the agency.
Thereafter, the software periodically and regularly
visits the portal to see whether it can find new
information assets or glean new metadata information
about information assets it previously indexed. Each
time, the software also downloads the data file(s)
associated with the information asset and tries to
extract additional metadata insights from the
downloaded data. For example, if one of the data files
was downloaded is in CSV or XLS format, the
software knows how to convert column names,
number of rows, and some statistics about the
of the information asset.
The software crawled only through the data.json
catalogs that agencies published on open government
data portals, as this is the only mandatory catalog that
the OMB instructs agencies to use. We admit that
agencies can and do publish important information
5
The number of active federal agencies is a topic of debate. We
used the list generated by the 2013 US Manual:
http://www.usa.gov/Agencies/Federal/Executive.shtml
 assets not through the data.json catalog. Still, we Table 3 – Assets and components by US
argue that there is good (even if incomplete) value in federal agencies
measuring agencies against the one mandatory open Agency Open Data Portal Assets Components
Department of Commerce (DOC) www.doc.gov/data.json 20488 21453
government data catalog that the Obama Environmental Protection Agency www.epa.gov/data.json 3285
4856
Administration instructed agencies to use. This paper (EPA)
Department of Transportation www.dot.gov/data.json 1654
is a starting point in our long research voyage to (DOT)
4621
study the information assets that agencies have Department of Health & Human www.hhs.gov/data.json 1633
5569
published since 2009. The most important and Services (HHS)
Department of Justice (DOJ) www.justice.gov/data.json 769 834
lowest-granular information in this corpus is the rich Department of Labor (DOL) www.dol.gov/data.json 364 592
metadata descriptions that agencies publish along Department of Agriculture www.usda.gov/data.json 331
734
(USDA)
with the data on their open government portals. USAID www.usaid.gov/data.json 234 421
To the best of our knowledge, not a single central Department of Veteran www.va.gov/data.json 227
670
Administration (VA)
repository exists today for scholars studying Department of Energy (DOE) www.energy.gov/data.json 225 644
agencies’ release of datasets on individual agency Department of State (DOS) www.state.gov/data.json 113 110
open government portals. Hence, the research General Services Administration www.gsa.gov/data.json 110
179
(GSA)
technique that we propose in this paper could also Department of Treasury www.treasury.gov/data.json 95
322
help researchers to systematically harvest information (Treasury)
National Archives (Archives) www.archives.gov/data.json 60 94
directly from individual agency open government Department of Housing and Urban www.hud.gov/data.json 56
105
portals in other countries or other levels of Development (HUD)
governments (e.g., municipal, county, state and Institute of Museum and Library www.imls.gov/data.json 40
171
Services (IMLS)
territory). Office of Personnel Management www.opm.gov/data.json 32
160
To address our key research questions regarding (OPM)
Nuclear Regulatory Committee www.nrc.gov/data.json 31
agencies’ compliance with open government data (NRC)
58
policy and the challenges and barriers to National Aeronautics and Space www.nasa.gov/data.json 25
82
implementing this policy, we examined key attributes Administration (NASA)
National Transportation Safety www.ntsb.gov/data.json 21 8
in our corpus to investigate this policy in four Board (NTSB)
19
domains: quality of metadata, time, gatekeepers, and Department of Homeland Security www.dhs.gov/data.json
9
2
0
access metadata attributes. In this exploratory (DHS)
National Science Foundation www.nsf.gov/data.json 2
research we found discrepancies between the open (NSF)
10 1
government data policy declarations and its Consumer Financial Protection www.consumerfinance.gov/ 1
7
execution. Next, we will discuss these findings. Bureau (CFPB) data.json
Department of Defense (DOD)11 www.defense.gov/data.json 0
Securities and Exchange www.sec.gov/data.json 0
12
5. Findings Committee (SEC)
Total: 29798 41,702
When examining the open government portals of
the 473 federal agencies in the US, we discovered
that only 25 agencies (5% of the total number of
agencies) adhere to the mandatory US federal open
data 2.0 standard (www.[agency].gov/data.json) and
have an open government portal. Three of these 8
In a few cases, such as the NTSB open government portal, we
agencies (DHS, DOD, SEC — see table 3) do not discovered fewer components than assets because the agency did
maintain information as mandated by the law and not provide a URL where the actual data could be downloaded, or
6 provided a malformed URL.
specified by the US federal open data 2.0 standard . 9
We failed to read and register information assets from DHS’
Therefore, the analysis below relies on the data data.json catalog, because DHS uses case-insensitive field labels
published by the remaining 22 agencies that did (e.g., “Title” instead of “title”). Similar to other Enterprise Data
partially comply with this standard. These 22 Warehousing (EDW) projects, we neither have the resources nor
agencies have published 29,798 information assets the motivation to “fix” the data-entry errors of individual agencies.
7 Wherever and whenever our software detected such problems, we
(41,702 components ) as shown in Table 3 below: contacted the agency and asked to fix its errors.
10
The NSF displays more assets in its data.json catalog than
registered here. However, because the metadata was malformed
and not in compliance with the technical specifications of the
data.json catalog, the software failed to read.
11
We could not read and register data from DOD’s data.json
catalog because DOD “invented” new non-standard JSON literal
6 types (e.g., “TRUE” instead of “true”).
Description of the metadata fields in data.json standard can be
12
found here: http://project-open-data.github.io/schema/ We could not read and register data from SEC’s data.json
7 catalog because SEC uses the XML standard instead of the
A “component” is a single or multiple files associated with a
single “information asset.” mandatory data.json standard.

2215

Authorized licensed use limited to: North Carolina Central University. Downloaded on November 27,2021 at 05:17:46 UTC from IEEE Xplore. Restrictions apply.
 5.1. Poor Metadata 5.2. Outdated Data
The first — and most significant finding —
concerns the poor, at times incomprehensible, quality Questions that open government data scholarship
of the metadata that agencies published to describe has attempted to answer are: What type of data is
their information assets. Interestingly, a specific being disclosed to the public? How old is the data?
metadata field (“dataQuality”) exists for agencies to When was the data created and how often is it
capture whether a given asset meets the agency’s updated? What is the declared frequency of data
Information Quality Guidelines (true/false). Yet updating and do government officials adhere to this
agencies did not provide this information for 87.72% frequency in reality? By using our corpus of data we
of all published information assets. When they did were able to provide initial answers for the first time.
report on it, they chose to mark their data as being of Figure 1 shows that in 21.43% of cases, agencies
good quality for 99.48% of the published assets. do not report the date when the information asset was
Agencies also ignored the recommendation of the US originally created; however, most agencies report
federal open data 2.0 standard to include in their open when the data was last updated (97.11%). A big
data portals metadata about information assets that portion of the data (60%) was created after 2009, the
they currently cannot release — 98.50% of all assets year the OGD 1.0 initiative was launched in the US.
are designated with an access level of “public” with According to Figure 1, 48% (14,316) of open
only 1.5% of assets (445 of 29,978 assets) marked as government data information assets were created in
“restricted public,” “non-public,” or not marked with 2011.
an access level.
The handful of partially complying agencies did
not provide other critically important metadata
information. For example, 70.48% of all the
information assets (21,002 of 29,798 assets) are not
associated with any program code (i.e., these codes
are the main vehicle through which Congress
provides funding for agencies to execute various
programs). Agencies did not provide even a single
keyword to tag 55.1% of all assets (16,419 assets).
Keywords are important as they help users to identify
assets’ value and relevance. Likewise, 87.63% of all
the assets are not associated with a “category” (i.e., a
category such as “Transportation” or “Health” is the
main thematic subject of the dataset while keywords
are more specific tags that help users discover assets.
Both “category” and “keywords” must be
comprehensible to technical and non-technical users). Figure 1 – Information assets created and
However, some good news exists: Agencies last updated
marked about 82% of all their information assets with
reasonable spatial bounds (i.e., the range of spatial However, in order to understand whether federal
applicability of a dataset including named places such agencies are compliant with the spirit of the OGD
as “New York City” or latitudinal and longitudinal policy, one must examine also agencies’ commitment
13
coordinates to identify the geographical location that to keep the data updated and fresh. Figure 2 below
the information asset describes). Agencies also shows that around 90% of all assets (26,732 assets)
marked about 80% of all the information assets with have not defined the frequency that the information
reasonable temporal bounds (i.e., bounds defining the asset must be updated. The frequency field is either
start and end dates of applicability for the data). not reported (left blank), or defined as “irregular,”
Remarkably, more than 97% of all the information “not planned or not scheduled” or “as needed.”
assets are tagged with a bureau code that empowers
end users to identify correctly which sub-unit/office
within the parent agency is responsible for the
creation of the information asset.
13
Unfortunately, currently we have only the last update of
information assets and not the number and dates of every update.
In the future the software will be able to track the updates as well.

2216

Authorized licensed use limited to: North Carolina Central University. Downloaded on November 27,2021 at 05:17:46 UTC from IEEE Xplore. Restrictions apply.

Figure 2 – Declared frequency update
Next, we examined agencies based on their
commitment to update the data they previously
published. As explained above, for more than 90% of
the information assets, agencies refused to commit to
a time interval such as “daily” or “weekly.” In fact,
agencies provided several creative values under the
“update frequency” field, such as “completely
irregular,” “notPlanned” and “None Scheduled.”
Still, we discovered 581 information assets that
contained a commitment to update these assets in
defined interval such as “weekly,” “monthly” or
“annually.” We discovered that agencies have failed
to keep their promise to update the data regularly in
32% of these cases.
5.3. Who Discloses Public Data?
The US federal open data 2.0 standard requires
the provision of the name of the contact person who
uploaded the asset. However, 76.3%% of all the
information assets (22,868 of 29,978 assets) do not
display the name of the person who uploaded the
asset or, alternatively, contain only the name of the
sub-unit responsible for the asset (such as “United
States Department of Justice. Bureau of Justice
Statistics”).
When examining the data, we find that 555
individual gatekeepers are responsible for the
disclosure of public data in US federal agencies. We
detected these gatekeepers by studying and analyzing
the metadata “author” of each information asset. The
“author” information was retrieved by the software
solution we used. We believe the “author” files
contain important information (that, in turn, we used
to run our gatekeeping analysis) because, often,
government officials upload information assets using
their own work terminals and software that
automatically registers their credentials. In other
words, even though we have yet to interview some of
2217
Authorized licensed use limited to: North Carolina Central University. Downloaded on November 27,2021 at 05:17:46 UTC from IEEE Xplore. Restrictions apply.
these individuals and survey the others, we are
confident that our analysis below relies on the
objective and daily activities of all the agency
officials that were engaged with uploading and
updating OGD information assets since 2009.
We noted immediately that there is a skewed
distribution in the amount of data disclosed by these
gatekeepers (see Figure 3). The role of gatekeepers
varies not only in the amount of public datasets they
disclose to the public, but also in the number of sub-
agencies these gatekeepers work with. We will
examine the endemic problems and opportunities of
current open government policy by comparing two
government officials responsible for uploading and
describing information assets on behalf of their
agencies. Ms. Gina Pearson is the Assistant
Administrator for Communications in the Department
of Energy (DOE) and has served in this role since
2006. She uploaded metadata for about 135 assets on
a single day (March 12, 2014) on behalf of various
DOE units. While doing so, Ms. Pearson used 42
keywords to describe the 135 information assets she
uploaded. Using Nahon’s network gatekeeper
terminology [40], Ms. Pearson is a traditional
gatekeeper whose influence in terms of sharing
information rarely traverses outside the boundaries of
her department (i.e., the DOE).
Figure 3: Power law of open government
data gatekeepers
In stark contrast, Mr. David Parrish, a 35-year
geographic information systems (GIS) veteran in the
EPA, was energetic about disseminating his agency’s
GIS data long before open government policy was
adopted in 2009. Years before www.data.gov was
launched, Mr. Parrish was working to release and
share information among units inside the EPA, with
other agencies, across levels of government, and with
research institutes and corporations outside
government. Our data shows that Mr. Parrish’s
 earliest updated asset is dated in April 1981 and his
latest one is dated in February 2014. Unsurprisingly,
Mr. Parrish enthusiastically adopted the open
government data program and most of the
information assets he uploaded were created after
2009, the birth year of the OGD. We did not
interview Mr. Parrish (we plan to do so as part of our
follow-up research report that will combine the
quantitative measurements we analyze in this paper
with more qualitative measurements that rely on
interviews and surveys of OGD officials such as Mr.
Parrish). However, it is important to remember that
Mr. Parrish’s boss, the CIO of the EPA, was one of
the two CIOs nominated by President Obama in 2009
to design, build and unleash www.data.gov (on May
21, 2009). Obviously, the CIO of the EPA was vested
personally in the success of this pet technology
project assigned to him by President Obama.
Therefore, we hypothesize here that Mr. Parrish
benefited enormously from the enthusiastic support
of his boss — the CIO of the EPA.
Most importantly, our corpus shows that Mr.
Parrish uploaded 582 open government data assets on
behalf of more than 50 internal EPA units and sub-
units as well as on behalf of 10 federal agencies
(BIA, BLM, BTS, DOE, NGA, NOAA, USACE,
USDA, USFWS, USGS), the agencies of 7 states
(Alaska, Arkansas, California, Hawaii, Nevada, New
Jersey, New York), four research
institutions/networks (CIESIN, Columbia Project,
ESRI, NGCC), one city (New York City), one county
(Morris County), and one corporation (TeleAtlas).
Mr. Parrish used 385 distinct keywords to describe
the 582 assets he uploaded.
In Nahon’s terminology, Mr. Parrish is a
boundary-spanner always seeking opportunities to
disseminate open data information assets deeply and
extensively inside his agency, across organizational
walls in government, and between government and
other sectors [40][41]. Regrettably, we find very few
boundary-spanners in our corpus. Without them, no
open data policy will reach wide and far. Possibly the
greatest opportunity to unleash the potential of the
US federal open data 2.0 standard is to identify,
nurture, and support a significantly large group of
boundary-spanners such as Mr. Parrish of the EPA.
5.4. Hard to Access
As explained above, the 29,978 information assets
whose metadata we analyzed contained 41,702
components (see footnote 7 for a definition of a
“component” and its relationship to “information
asset”). The software we used downloaded the 41,702
components, analyzed them, and registered in our
corpus additional metadata information about the
2218
Authorized licensed use limited to: North Carolina Central University. Downloaded on November 27,2021 at 05:17:46 UTC from IEEE Xplore. Restrictions apply.
information assets associated with these components.
We analyzed this metadata and discovered that 82%
(33,862 of 41,702) of all the components were not
machine-processable as required by the OGD 2.0
policy. Put differently, we discovered that it did not
matter much if end users (after much trouble)
managed to navigate to one of the data ships in the
OGD ocean. In more than 80% of the cases, such
navigation efforts end in frustration when end users
discover that the “data” they seek is saved in a non-
machine-processable file type such as PDF, HTML,
JPG or TIFF. In addition, we discovered that 6% of
all the download URLs (i.e., the URL provided by
the agency to empower the end-user to download the
actual data) are broken links.
6. Discussion: From Released Data to
Open Government
One important limitation of this study is that the
software that we used in this article captures only a
small (though very interesting) part of the
information-release strategies of US federal agencies.
Therefore, our ability to answer broad questions such
as “are US federal agencies more transparent?” based
on the data we analyzed is very limited.
Still, even within the boundaries of our limited
data, we discovered interesting findings that
contribute something original to the growing body of
literature that describes the information release
strategies of public-sector agencies. For example,
only 5% of US federal agencies met the minimal US
federal open data 2.0 standard requirement (i.e.,
publish a data.json catalog on their servers to list
inventories of their information assets). Most of the
data that the small subset of complying agencies
published suffers from poor quality of metadata,
outdated data, a tiny number of gatekeepers who
publish the vast amount of data, and accessibility
issues. Our analysis clearly demonstrates the
differences between the four modes of information
states: released data, accessible data, open data and
open government. Most of the data disclosed to the
public is incomprehensible. Some of the data released
is not even accessible. For example, citizens are
frustrated by broken links, non-machine-processable
data such as HTML information, or data that
completely lacks descriptions. The slogan “if we
build it, they will come” that represents the belief that
the release of large volumes of data will increase
transparency and participation is here shown to be a
naïve one.
Even when agencies make their data accessible,
there is no guarantee that the released information is
 truly open. The little data that is open,
comprehensible and accessible generates even more
troubling questions: Does such data promote
transparency and governmental accountability? Does
the data contribute to the strengthening and
improvement of the government, including the values
of public participation, citizens’ empowerment and
the democratic ideal? Our analysis proposes that the
journey from the initial disclosure of public data to
implementing open government principles is long.
Our study examined for the first time in a
systematic, quantitative way the status of open
government data in the US by focusing on the
disclosure of data by US federal agencies as
mandated by the dictates of the US federal open data
2.0 standard. Our findings suggest that most US
federal agencies largely do not comply with this
standard while only 25 agencies partially and weakly
comply with it. Most information assets that are
published are not updated, display partial data
descriptions and sometimes even broken links to the
data, and are hard to search and find. Therefore, for
now, and five years after the birth of the original
open government data policy, the US open
government data movement is metaphorically
comparable to a vast ocean with sparsely scattered
data ships; a majority of these ships are lost at sea,
sailing sans target, purpose or captain.
7. References
[1] H. Yu and D. Robinson, “The New Ambiguity of
‘Open Government,’” UCLA Law Rev. Discourse,
vol. 59, pp. 178–208, 2012.
[2] Office of Management and Budget (OMB), “Open
Government Directive,” Washington D.C.,
Memorandum No. M-10-06, Dec. 2009.
[3] A. Peled, “When Transparency and Collaboration
Collide: The USA Open Data Program,” J. Am. Soc.
Inf. Sci. Technol., vol. 62, no. 11, pp. 2085–2094,
2011.
[4] B. S. Noveck, Wiki Government: How Technology
Can Make Government Better, Democracy Stronger,
and Citizens More Powerful. Brookings Institution
Press, 2009.
[5] Open Knowledge Foundation, “The Opne Data
Handbook Documentation,” 2012.
[6] B. Noveck, Testimony of Dr. Beth S. Noveck Before
the Standing Committee on Access to Information,
Privacy, and Ethics of the Canadian Parliament".
House of Commons Canada, 2011.
[7] E. Barry and F. Bannister, “Barriers to Open Data
Release: A View from the Top,” presented at the
European Group for Public Administration,
Edinburgh, 2013.
[8] M. Janssen, Y. Charalabidis, and A. Zuiderwijk,
“Benefits, Adoption Barriers and Myths of Open Data
and Open Government,” Inf. Syst. Manag., vol. 29,
no. 4, pp. 258–268, Sep. 2012.
2219
Authorized licensed use limited to: North Carolina Central University. Downloaded on November 27,2021 at 05:17:46 UTC from IEEE Xplore. Restrictions apply.
[9] G. Lee and Y. H. Kwak, “An Open Government
Maturity Model for social media-based public
engagement,” Gov. Inf. Q., vol. 29, no. 4, pp. 492–
503, Oct. 2012.
[10] L. B. Bingham and S. Foxworthy, “Collaborative
Governance and Collaborating Online: The Open
Government Initiative in the United States,” presented
at the Converging and Conflicting Trends in the
Public Administration of the US, Europe, and
Germany, 2012.
[11] J. C. Bertot, P. McDermott, and T. Smith,
“Measurement of Open Government: Metrics and
Process,” in 2012 45th Hawaii International
Conference on System Science (HICSS), 2012, pp.
2491–2499.
[12] S. J. Piotrowski, Governmental Transparency in the
Path of Adminstrative Reform. Albany: State Univ of
New York Pr, 2007.
[13] R. Gelman, “The Foundations of United States
Government Information Dissemination Policy,” in
Public Sector Information in the Digital Age –
Between Markets, Public Management and Citizens’
Rights, G. Aichholzer and H. Burkert, Eds.
Northhampton, MA: Edward Elgar, 2004, pp. 123–
136.
[14] S. Holden and P. Fletcher, “The Virtual Value Chain
and E-Government Partnership: Nonmonetary
Agreements in the IRS E-Files Program,” in
Handbook of public information systems, D. Garson,
Ed. Boca Raton, FL: Marcel Dekker, 2005, pp. 369–
387.
[15] Office of Management and Budget (OMB),
“CIRCULAR NO. A-130 Revised.” 28-Nov-2000.
[16] B. Obama, “The Freedom of Information Act
(FOIA),” Washington D.C., Memorandum, Jan. 2009.
[17] P. McDermott, “Building open government,” Gov. Inf.
Q., vol. 27, no. 4, pp. 401–413, Oct. 2010.
[18] Executive Office of the President, “Open Data Policy
- Managing Information as an Asset,” Washington
D.C., Memorandum No. M-13-13, May 2013.
[19] Office of Management and Budget (OMB),
“Supplemental Guidance on the Implementation of
M-13-13 ‘Open Data Policy – Managing Information
as an Asset.’” 2013.
[20] R. P. Lourenço, “Data Disclosure and Transparency
for Accountability: A Strategy and Case Analysis,”
Inf. Polity, vol. 18, no. 3, pp. 243–260, Jul. 2013.
[21] A. Zuiderwijk, M. Janssen, S. Choenni, R. Meijer, and
R. S. Alibaks, “Socio-technical Impediments of Open
Data,” Electron. J. E-Gov., vol. 10, no. 2, pp. 156–
172, 2012.
[22] The White House, “Open Government Initiative -
Around the Government,” 2010. .
[23] J. Wonderlich, “Obama’s Open Government
Directive, Two Years On,” Sunlight Foundation, 07-
Dec-2011. .
[24] Public Accounts Committee, “Implementing the
transparency agenda,” 2012. .
[25] T. Van Den Broek, B. Kotterink, N. Huijboom, W.
Hofman, and S. Van Grieken, Open Data Need A
 Vision Of Smart Government: Roadblocks To A Pan
European Market For PSI Reuse. 2011.
[26] J. Harper, “Grading the Government’s Data
Publication Practices,” Policy Anal., vol. 711, pp. 1–
43, Nov. 2012.
[27] G. Bass, D. Brian, M. Fuchs, A. Schwartz, P.
McDermott, E. Miller, and A. Weismann, “Letter
Encouraging the Administration to Improve Its Open
Government Efforts,” 2010. .
[28] A. C. Thurston, “Trustworthy Records and Open
Data,” J. Community Inform., vol. 8, 2012.
[29] J. Harper, “Government Spending Transparency:
‘Needs Improvement’ Is Understatement,”
CATO@LIBERTY, 2011. .
[30] R. J. Cole, “Some Observations on the Practice of
Open Data As Opposed to Its Promise,” J. Community
Inform., vol. 8, 2012.
[31] T. Davies and Z. A. Bawa, “The Promises and Perils
of Open Government Data (OGD),” J. Community
Inform., vol. 8, 2012.
[32] V. Kundra, “From Data to Apps: Putting Government
Information to Work for You,” The White House
Blog, 2011. .
[33] F. Bannister and R. Connolly, “The Trouble with
Transparency: A Critical Review of Openness in e-
Government,” Policy Internet, vol. 3, no. 1, pp. 1–30,
Feb. 2011.
[34] A. Schellong and E. Stepanets, “Unchartered waters –
the State of Open Data in Europe,” Wiesbaden,
Germany, Jan. 2011.
[35] UK Comptroller and Auditor General, “Implementing
Transparency: Cross-Government Review,” 2012. .
[36] M. Biddick and W. Kash, “2014 Federal Government
IT Priorities,” Washington, D. C, 2013.
[37] A. Zuiderwijk, K. Jeffery, and M. Janssen, “The
Potential of Metadata for Linked Open Data and its
Value for Users and Publishers,” JeDEM - EJournal
EDemocracy Open Gov., vol. 4, no. 2, pp. 222–244,
Dec. 2012.
[38] A. Peled, “Re-Designing Open Data 2.0,” JeDEM -
EJournal EDemocracy Open Gov., vol. 5, no. 2, pp.
187–199, Dec. 2013.
[39] B. Obama, “Making Open and Machine Readable the
new default for Government Information,”
Washington D.C., Executive Order, May 2013.
[40] K. Barzilai-Nahon, “Toward a Theory of Network
Gatekeeping: A Framework for Exploring Information
Control,” J Am Soc Inf Sci Technol, vol. 59, no. 9, pp.
1493–1512, Jul. 2008.
[41] K. Nahon and J. Hemsley, Going Viral, 1 edition.
Polity, 2013.

2220

Authorized licensed use limited to: North Carolina Central University. Downloaded on November 27,2021 at 05:17:46 UTC from IEEE Xplore. Restrictions apply.