AI AND GOVERNMENT
Editor: Vassilios peristeras, European Commission, vassilios.peristeras@ec.europa.eu
Open Government Data:
A Data Analytics Approach
John S. Erickson, Amar Viswanathan, Joshua Shinavier, Yongmei Shi,
and James A. Hendler, Rensselaer Polytechnic Institute
I
n December 2010, the International Open Gov-
ernment Dataset Search (IOGDS) team at the
Tetherless World Constellation (TWC) at Rensselaer
Polytechnic Institute (RPI) embarked on a project
to discover, document, and analyze open data
catalogs published by governments at various lev-
els around the world.1 By early 2013, the IODGS
project had accumulated descriptive metadata for
more than 1,022,787 datasets from 192 catalogs
in 24 languages, representing 43 countries and
international organizations.
RPI’s aggregate catalog, implemented using the
Resource Description Framework (RDF) and pub-
lished using both a public SPARQL endpoint and a
faceted user interface, has proven to be a valuable
tool for gaining insight into the nature of open
government data publication. Here, we discuss
what our team has learned about international
government data publication trends and tenden-
cies through the application of data analytics and
data visualization to this metadata collection.
Open Government
Data Publication: A Review
Motivated by the first Obama Administration’s
transparency initiatives, in May 2009 the US
launched the Data.gov Web portal with a catalog
of 47 datasets containing previously unreleased
government data. During its first year, Data.gov
grew to more than 250,000 datasets, inspired hun-
dreds of applications and services, and was seen as
the flagship of the worldwide movement toward
open government data publication. Other govern-
ments followed in rapid succession, and in the next
few years open government sites for countries, mu-
nicipalities, cities, and others went online.
The significant growth in number and size of
open government data catalogs since 2009 has
been made possible by the emergence of an open
government data ecosystem consisting of policy
SEpTEMbEr/ocTobEr 2013 1541-1672/13/$31.00 © 2013 IEEE
Published by the IEEE Computer Society
IS-28-05-gov.indd 19
makers, agencies (as providers and consumers),
data experts, independent software developers
and service providers, academia, and citizen stake-
holders. The publication of widely varied data has
inspired a wide assortment of applications and
services; provided essential data for journalists,
bloggers, and activists; and has fueled academic
research. In turn, demand from stakeholders has
increased the quality and variety of this data,
and has encouraged the publication of thousands
of  datasets as readily consumed linked open
government data.2
Building a Million-Dataset Catalog
Starting in 2010, countries published open gov-
ernment data catalogs using a range of platforms
and cataloging approaches. Our project recog-
nized these catalogs as potentially valuable sources
of key data, providing names, descriptions, and
URLs of datasets from many countries, if only the
contents of those catalogs could be collected and
analyzed. Governments seldom publish data cat-
alogs using uniform data models, much less as
machine-readable RDF following linked data prin-
ciples.3 To generate a uniform aggregate catalog,
our team developed a semi-automated process that
included manual data portal identification; cata-
log and dataset metadata identification; per-catalog
customization of metadata harvesting tools; and
automated linked data conversion and publication
on a public SPARQL endpoint based on an existing
laboratory infrastructure.4 A novel faceted browser
developed for the Semantic eScience Framework
(SeSF)5,6 project was adapted to provide a highly
efficient faceted browse and search experience for
the user.
IOGDS didn’t consider certain other character-
istics of open government data publication that
might be of particular interest to practitioners. For
example, it would be useful to have greater detail
regarding the fi le formats in use, giving us deeper
19
17/12/13 4:38 PM

Figure 1. Word cloud depicting the keyword analysis of Data.gov. Excluding the
geodata catalogs gives us a more representative picture of the diversity of data.
insight into the extent of machine-
readable data publication.7 Potential
adopters might be interested in the
relative penetration of commercial
(for example, Socrata; http://socrata.
com), open source (such as Open Gov-
ernment Platform; http://ogpl.gov.
in, and CKAN; http://ckan.org), and
ad hoc data platforms. It would also
have been fascinating to have gath-
ered detailed time series data docu-
menting the emergence of catalogs
around the world. All of these details
exceeded our capabilities, given the ir-
regularity of deployed platforms and
the resulting challenges of gathering
metadata. Finally, while IOGDS has
focused on the publication of data it-
self, we recognize that many govern-
ment data portals have been designed
to sustain communities of stakehold-
ers, and thus also provide support for
related applications and services.
The Analytics of Government
Data Publication
The IOGDS catalog (http://logd.tw.
rpi.edu/page/international_dataset_
catalog_search) has served as an ob-
servatory for exploring the diversity
20 www.computer.org/intelligent IEEE INTELLIGENT SYSTEMS
IS-28-05-gov.indd 20
of datasets published on the Web by
political entities around the world.
Here, we provide insights into this
catalog based on analytics performed
across the collected metadata.
Catalogs Published by Individual
Countries
During our collection period, the US
was the global leader in publishing
(453,859 datasets), followed by France
(353,394), Canada (179,131), the UK
(12,131), and Spain (10,076). These
statistics include geographic datasets
of various kinds, which account for
the largest subset of entries.
Languages across All Catalogs
Our analytics found a total of 24 lan-
guages represented across all catalogs in
the IOGDS collection. English is by far
the most prominent language (98 cata-
logs/652,176 datasets), since most of
the open government data published
through late 2012 came from English-
speaking countries. Other notable lan-
guages included French (19 catalogs/
528,153 datasets), Spanish (18 catalogs/
8,444 datasets), Italian (14 catalogs/2,256
datasets), and German (12 catalogs/1,584
datasets). Nineteen languages were rep-
resented in the remaining 31 catalogs.
Catalog Contents by Keyword
To identify trends in data publication,
we studied keywords associated with
datasets and categories describing cat-
alogs. We saw that certain data cate-
gories tend to dominate the keyword
frequency analysis for some catalogs;
for example, geographic ­datasets ac-
count for more than 441,348 datasets
in the US Data.gov. Keyword analy-
sis of Data.gov with geodata cata-
logs excluded (reducing the count to
4,826 datasets) gives us a more rep-
resentative picture of the diversity of
data, as depicted by the word cloud
in Figure 1.
Catalogs published by municipal
governments provide a clue as to the
priorities of city governments and their
stakeholders. Analysis of keyword
sets harvested from catalogs such as
New York City (http://nycopendata.
socrata.com), San Fransisco (http://
data.sfgov.org), and Edmonton (http://
data.edmonton.ca) helped our team see
both the similarities and differences of
these governments.
For example, the word cloud in Fig-
ure 2 displays the top keywords from
859 datasets published by New York
City (http://nycopendata.socrata.com).
Clearly, the city has placed heavy em-
phasis on the publication of datasets
containing location data and datasets
relevant to education and community
services.
IOGDS has enabled us to study the
publication tendencies of agencies
within governments, as well as inter-
national organizations. For example,
Figure 3 displays the top keywords
from the 1,405 datasets published by
the US government’s Medicare pro-
gram (http://data.medicare.gov).
An examination of keywords from
10,678 datasets published in the UK
government catalog (http://data.gov.uk)
17/12/13 4:38 PM
 shows a rich diversity of data covering
many sectors that it shares with other
countries’ catalogs (see Figure 4). The
metadata also suggests heavy use of cer-
tain generic keywords such as “trans-
parency” and “disclosure”—which
might be tied to UK policies, but such
keywords were uncommon to the US or
other countries’ catalogs we examined.

Analyzing Dataset Categories

The IOGDS category metadata char-
acterizes subsets of datasets across
catalogs. Analysis of the distribution
of categories across countries gives
us some insight into the priorities of
governments as they publish their
data. In Table 1, we use categories to
summarize the publication similari-
ties and differences between selected
governments.
Making Open Government
Data Discoverable and
Usable
The amount of work required to col-
lect the information in this ­catalog
was high, and maintaining it would
be a significant effort. To be able
to track this kind of information
on an ongoing basis, and to bet-
ter compare datasets across govern-
ments (as well as to provide better
search tools), we make the following
recommendations.
• The importance of standard vocab-
ularies. Datasets regardless of their
coverage are inherently opaque and
require external metadata to facili-
tate cataloging and discovery. The
insights our team gained by per-
forming data analytics across the
aggregate IOGDS catalog were
just a glimpse of what’s possible
through the widespread adoption
of metadata standards for pub-
lishing catalogs of data. We look
forward to the increased imple-
mentation of new standards, such
Figure 2. Word cloud displaying the top keywords from 859 datasets published by
New York City. The city has placed heavy emphasis on the publication of datasets
containing location data and datasets relevant to education and community services.
as World Wide Web Consortium’s government data providers and
Data Catalog Vocabulary (W3C their partners will lead to radically
DCAT),8 in the near future and increased accessibility for the stake-
are  especially encouraged by the holder community.
formal acceptance of DCAT within • The need for machine-readable for-
the European Community.9 mats for publishing metadata. We’re
• The value of microdata markup witnessing a move towards distrib-
and large-scale indexing. Metadata uted publication of datasets within
is of little practical use for large- governments as agencies begin to
scale discovery unless it has been manage their own catalogs and
exposed on the Web in ways that portal-wide aggregation happens
­
enable efficient indexing by search only virtually. It’s therefore critical
engines and other applications. for governments to embrace commu-
Schema.org (http://schema.org) is nity-developed technical standards
a search engine company-led, com- for the machine-readable expression
munity-driven effort centered on of catalog contents,7 so that their da-
the publication of consensus vo- tasets can be included in local, feder-
cabularies for expressing a website’s ated, and global data catalogs.
structured data through on-page
markup, enabling search engines
to better parse the information on
webpages and provide richer search The application of data analyt-
results. The widespread adoption of ics to the TWC RPI IOGDS catalog
the Schema.org Dataset extension has  enabled a kind of “world tour”
(http://schema.org/Dataset) by open of the international open ­government

September/october 2013 www.computer.org/intelligent 21

IS-28-05-gov.indd 21 17/12/13 4:38 PM

 scale and higher frequency using
metadata collected from the world’s
catalogs much more efficiently. This
will be made possible through fed-
erated mechanisms enabled by new
standards and best practices that
are emerging with the evolution of a
more mature open government data
ecosystem.

Acknowledgments
This work has been made possible by a gen-
erous gift to the Tetherless World Constella-
tion at Rensselaer Polytechnic Institute from
Microsoft Research.

Figure 3. Word cloud showing the top keywords from the 1,405 datasets published
by the US government’s Medicare program.
Figure 4. Word cloud examining keywords from 10,678 datasets published in the UK
government catalog (http://data.gov.uk).
data movement. Our analysis of the
IOGDS data has shed some light on
the coverage, trends, and diversity of
published data around the world.
References
In the future, the adoption of bet-
ter standards will allow applica-
tions and services to be able to re-
peat our data analytics at larger
1. J.S. Erickson et al., “TWC Internation-
al Open Government Dataset Catalog,”
Proc. 7th Int’l Conf. Semantic Systems,
ACM, 2011, pp. 227–229.
2. L. Ding, V. Peristeras, and M. Hausen-
blas, “Linked Open Government Data,”
IEEE Intelligent Systems, vol. 27, no. 3,
2012, pp. 11–15; http://bit.ly/16YYb7s.
3. T. Berners-Lee, “Linked Data,” W3C
Design Issues, 27 July 2006; http://bit.
ly/cwflPW.
4. L. Ding et al., “TWC LOGD: A Portal
for Linked Open Government Data
Ecosystems,” Web Semantics: Science,
Services and Agents on the World Wide
Web, vol. 9, no. 3, 2011; http://bit.
ly/16tmY9q.
5. E. Rozell, et al., “A Framework for
Integrating Oceanographic Data
Repositories,” Proc. AGU Fall Meeting
2010, Am. Geophysical Union, 2010;
http://bit.ly/1dLdxFc.
6. E. Rozell, Extensible User Interface
Framework for Faceted ­Browsing
­Applications, master’s thesis,
­Rensselaer Polytechnic Inst., 2012;
http://bit.ly/16EnnpG.
7. J. Hendler and T. Pardo, Open Govern-
ment Primer on Machine-Readability,
blog, 24 Sept. 2012; www.data.gov/
communities/node/116/blogs/76451.
8. F. Maali and J. Erickson, eds., Data
Catalog Vocabulary (DCAT), W3C.

22 www.computer.org/intelligent IEEE INTELLIGENT SYSTEMS

IS-28-05-gov.indd 22 17/12/13 4:38 PM

 Table 1. Summary (in %) of top data categories across catalogs for selected countries.

US UK Canada Australia Germany

Administrative and
political boundaries
(18.01)
Imagery and base maps
(12.87)
Transportation
networks (12.49)
Imagery Base Maps
Earth Cover (11.083)
Boundaries (9.38)
Geological and
geophysical (7.91)
Locations and geodetic
networks (7.85)
Inland water resources
(5.54)
Transportation (3.66)
Oceans and estuaries
(2.33)
Inland Waters (1.91)
Oceans (1.58)
Facilities and
structures (1.51)
Structure (0.87)
Climatology Meteorology
Atmosphere (0.44)
Geography and
­environment (0.20)
Elevation (0.1)
Biota (0.09)
Environment and
­conservation (0.06)
Health and Economics and i­ndustry Community (11.09) Air (11.27)
social care (10.53) (47.36)
Economy (4.07) Health and safety Geography (9.31) Water (11.06)
(12.34)
Children, education, Society and culture Government (8.79) Other (8.08)
and skills (3.67) (6.34)
People and places Nature and Business (8.79) Politics and
(3.43) environment (4.76) ­government (5.14)
Population (2.93) Labor (3.71) Environment (7.68) Economy (4.16)
Agriculture and Persons (3.24) Finance (6.89) Education and
environment (2.54) science (3.92)
Crime and justice (2.21) Agriculture (2.84) Sciences (6.38) Radiation (3.20)
Travel and transport Education and Society (6.18) Public administration,
(2.04) training (2.66) budget, and taxes (3.18)
Business and energy Transport (2.1) Industry (5.50) Traffic (3.18)
(2.01)
Parish (1.84) Government and Recreation (3.68) Labor market (2.94)
­politics (1.84)
Government (1.69) Impacts and environmental Culture (2.85) People, family, and
change (1.41) social affairs (2.94)
Health (1.53) Science and rechnology Health (2.33) Climate environment
(1.19) and nature (2.69)
Demographics (1.32) Information and Law (2.06) Basic data and
­communication (1.14) ­geosciences (2.20)
Labor market (1.32) Energy and greenhouse Transport (1.86) Building and living
gas (GHG) emissions (1.96)
(1.03)
Education (1.02) Law (0.86) Planning (1.66) Trade commerce (1.71)
Health (0.95) Business and economic General (1.58) Leisure culture and
development (0.48) tourism (1.71)
Employment and skills Census (0.48) Safety (1.42) Population growth (1.71)
(0.85)
Environment (0.79) Processes (0.33) Property (1.22) Demographics (1.47)
Transport (0.75) Development (0.25) Communication (1.14) Environment and
­climate (1.47)

org, 1 Aug. 2013; www.w3.org/TR/
vocab-dcat.
9. European Commission: Interop-
erability for European Public
­A dministrations (ISA), DCAT
­A pplication Profile for Data Portals
in Europe, 2 Sept. 2013; http://bit.
ly/19kaBwo.
Amar Viswanathan is a PhD student at the
Teth­erless World Constellation at Rensselaer Poly-
technic Institute. Contact him at kannaa@rpi.edu.
Joshua Shinavier is a PhD student in com-
puter science at the Tetherless World Con-
stellation at Rensselaer Polytechnic Institute.
Contact him at shinaj@rpi.edu.
James A. Hendler is the director of the
Rensselaer Institute for Data Explora-
tion and Applications (IDEA), the Tether-
less World Senior Constellation Chair, and
a member of the faculty in the Department
of Computer Science and the Department of
Cognitive Science at Rensselaer Polytechnic
Institute. Contact him at hendler@cs.rpi.edu.

John S. Erickson is the Director of Web Sci- Yongmei Shi is a research associate at the
ence Operations at the Tetherless World Con- Tetherless World Constellation at Rensselaer Selected CS articles and columns
stellation at Rensselaer Polytechnic Institute. Polytechnic Institute. Contact her at yong- are also available for free at
Contact him at erickj4@rpi.edu. mei.shi@gmail.com. http://ComputingNow.computer.org.

September/october 2013 www.computer.org/intelligent 23

IS-28-05-gov.indd 23 18/12/13 3:03 PM