Fleck Open Access Gov

This is a draft version. It will change and may be removed from ScribD.
Please do not cite without permission.
Peter Fleck
December 12, 2009
Opening Democracy: Sharing the Raw Ingredients of Policy

Peter Fleck
January 4, 2010
©[copyright] 2009 Peter Fleck
If the next Presidential administration really wants to embrace the potential of

Internet-enabled government transparency, it should follow a counter-intuitive
but ultimately compelling strategy: reduce the federal role in presenting important
government information to citizens.
Government Data and the Invisible Hand (Robinson, Yu, Zeller, & Felten, 2008)
There is no doubt as to how much the 2008 Obama campaign and Obama’s
presidential administration are embracing the Internet. More than a million people
signed up for campaign text messages on their cell phones; two million joined the
MyBO portal (My.BarackObama.com) where social networking merged with
volunteer work; and another 13 million signed up for regular email reports (Melber,
2008). By Election Day, twenty-five percent of Obama voters were linked to him and
each other through one of these networks (Melber, 2008).1 After the election in
1
Not all the network efforts were successful. The idea of announcing his vice-president choice via cell
phone text messaging was a failure in itself (sending ten million cell phone text messages at one time is
just not possible with today’s infrastructure) but a smart marketing move in harvesting all the cell
This is a draft version. It will change and may be removed from ScribD. Please do not
cite without permission.
November, the Obama tech team immediately set up Change.gov to document the
transition to power and to solicit ideas from the public (Jardin, 2008).2
These efforts were savvy organizing moves that utilized the power of the
Internet to aggregate information and categorize it for storage in a database. They
were a new generation marketing tool that allows citizens to engage in the
conversation and the debate.3
But President Obama foresees a much greater potential for the Internet and its
relationship to government. Before his election to the presidency—as Senator from
Illinois—he along with Oklahoma Senator Tom Coburn, sponsored the Federal
Funding Accountability and Transparency Act of 2006. On January 21, 2009—the
day of his inauguration—he issued the Transparency and Open Government
Memorandum, which states that his Administration “is committed to creating an
unprecedented level of openness in Government” and to disclosing “information
rapidly in forms that the public can readily find and use” (Obama, 2009). This echoes
his campaign technology and innovation plan, which proposes creating “a transparent
and connected democracy” to “integrate citizens into the actual business of
government” by “making government data available online in universally accessible
formats to allow citizens to make use of that data to comment, derive value, and take
action in their own communities” (Obama for America, ND). In a move in this
direction, on March 5, 2009 President Obama chose open data proponent Vivek
phone contacts (McConnell, 2008).

2
Since inauguration, the Change.gov site states the transition is over and invites us to join President
Barack Obama at Whitehouse.gov. However, as per good Internet practice, the original Change.gov
site was archived and is still available for browsing (Whitehouse.gov, 2008).
3
A good example of participating in policy debate occurred during the campaign in the summer of
2008 when Obama backed Bush's domestic-spying bill. Supporters used the social network portal to
protest the move and were able to get a direct response from Obama on the issue and also to place a
civil liberties argument on Obama’s agenda (Melber, 2008).
Fleck: Opening Democracy: Sharing the Raw Ingredients of Policy ©2009 Peter 2
Fleck
Kundra as his Chief Information Officer (CIO) (White House Office of the Press
Secretary, 2009). Kundra had been instrumental in opening up data when he was
Chief Technology Officer (CTO) for the District of Columbia. In Kundra’s first
conference call with the press, he said, “How do we make sure that the government is
about ‘We the People’ and that we engage citizens in terms of how their government
functions—holding government officials accountable in terms of making sure they
know where money is going throughout the public sector, ensuring that we have the
ability to run an open, transparent, participatory, and collaborative government”
(O’Brien, 2009).
The push to make federal government data available easily to the public—
open access—is quickening under the Obama administration but it didn’t start there.
The federal level has been grappling with this issue since before 1946 when the
Administrative Procedure Act became law. Before the Internet, disseminating public
information was a difficult and expensive task involving photocopying documents
and delivering them to the interested parties. This could involve visiting an office and
making a request or using the mail system. The Internet has given us new protocols to
provide almost direct access to data at a lower cost than providing the data through a
“brick-and-mortar” office. It has enabled us to be the clerks that gather the data. The
advantage for us in this scenario is the ability to “timeshift” our tasks to our schedule.
We don’t have to collect the data between nine and five and show up at an office; we
can now do it from 9 p.m. to 5 a.m via the Internet from our homes.
Brito (2007) noted that much publically available data is not available online
and what is usually is not in a “useful format.” He called for the government to
Fleck
“release public information online in a structured, open, and searchable manner”
(Brito, 2007). That premise is at the heart of this paper. Government at all levels must
begin the process of migrating the data that anchors policy to the Internet and provide
open access.
This paper presents the case for providing government data in “universally
accessible” or “structured” formats as the best way to integrate the most citizens in
the actual business of democracy and to promote government transparency. Brito
(2007) defines a “structured format” as “information… presented in a format that
allows a computer to easily parse and manipulate it.” In most cases this will mean a
CSV4 or XML5 format—directly from a database. Besides being the best way to
guarantee transparency, this open access to government data saves money in
deployment, informs the public to a higher degree, and provides a path to deep
engagement with our democracy. I include a short history of government
transparency legislation and I detail several groundbreaking projects in the area of
open access to government data. I’ll also describe the “ancient” methods of scraping
screens and republishing data.
The twentieth century idea of building more web pages to inform the
electorate will continue to be a necessary and important function but it is not the same
as providing the raw data behind government policy-making in a format that third
parties (researchers, students, journalists, small businesses, citizens) can slice, dice,
and analyze in a thousand ways. Web pages share with printed documents a static
4
CSV stands for “comma separated values.” It is a standard method of exporting and importing
database information in text files that does not rely on any particular software application or operating
system.
5
XML stands for “extensible markup language” and it is a global standard. It is meant to be read by a
machine (computer) and not a human.
Fleck
formulation based on “pushing” selected information to “consumers.” Federal CIO
Vivek Kundra, states that “If you look at government, what they've done historically
is they've just put up a website and they'll say this is Agency X. Unfortunately, if you
look at the traffic on those websites and you compare that traffic to a Facebook or a
Craigslist, it just pales in comparison and one of the things we need to start thinking
about is how do we put information in the right context” (O’Brien, 2009). Brito
(2007) says that the government should not try to offer “one best way” to use the data
but instead “allow myriad third parties develop innovative tools that utilize the data.”
According to Clift (2007), using open technologies like XML will “make it
easy to re-use public government data from many sources to create views and
searches that provide insight, understanding, and accountability.” This will give us
“‘Web 2.0’ interactivity built on top of government data by those outside of
government” (Clift, 2007). That last part about building “outside of government” is
important. Open-access activists have already proven that they will build the
interfaces to read the XML data. Brito (2007) states that as long as “information is
available in an open and structured format” developers can “mash” the data in
unpredictable ways and “highlight patterns that would be otherwise imperceptible in
the source data.”
Scraping Screens
As proof of the “outside of government” efforts of citizen open-access
activists, many informative web sites have been constructed using a method known as
“screen-scraping.” “Screen-scraping” involves writing a program that will read in a
web page and find the data in the page that you are looking for. Since web pages for
Fleck
the most part are unstructured documents to be read by humans with no underlying
standardized structure that can be easily parsed for data sharing, this is an immense
challenge. Yet incredible sites have been created using this method.6
One of the classic screen-scraping site using public data is ChicagoCrime.org
created by Adrian Holovaty in May of 2005.7 Holovaty (2005) describes it:
The site is a freely browsable database of crimes reported in Chicago. My scripts

collect data from the Chicago Police Department once every weekday. The site
slices and dices crime information in a ton of different ways, complete with a
wide assortment of Google Maps.
Holovaty’s scripts must grab data from the police site and determine what is relevant
for his “mashup” web application.8 He can only hope that the Chicago Police web
department doesn’t decide to change something on their site that could require
reprogramming of Holovaty’s scripts. The data on the web pages has no standard
structure of any kind. Imagine trying to blindly extract information from a particular
article in a magazine. How would you know when to turn the page? How would you
know which paragraph has the important information? Then repeat your process on
6
Lincoln Stein (2002) compares screen-scraping to “mediaeval torture.” Brito (2007) calls it “much
like the hand copying of texts by medieval monks” and provides this more detailed description:
“In essence, “screen-scraping” involves calling up the web page that displays the type of data the user
wishes to gather (for example, a senate roll call vote page), identifying the patterns apparent on the
page (such as where the bill title and number are displayed and which boxes correspond to the yeas and
nays), and then writing a computer script that will transfer data found in designated display positions to
the appropriate fields in a database. In many ways this is the digital equivalent of having to scan paper
copies of documents because, while the original may well be electronic in this case, it is the final user
display that is accessed and parsed into meaningful groupings. In short, it is an inefficient and often
inexact method.”
7
The web URL ChicagoCrime.org now redirects to EveryBlock.com, Holovaty’s new web project that
grew out of the crime site. EveryBlock is tracking fifteen cities. In addition to crime statistics, it tracks
building permits, restaurant inspections, news and blog entries about the city or block, and local photos
and restaurant reviews. Funding for EveryBlock is from the Knight News Foundation via the Knight
News Challenge (Holovaty, 2009). On June 30, 2009, Holovaty released the EveryBlock programming
source code as open source allowing any city to establish its own EveryBlock site (Brady, 2009).
8
A mashup is “a web page or application that combines data or functionality from two or more
external sources to create a new service. The term mashup implies easy, fast integration, frequently
using open APIs and data sources to produce results that were not the original reason for producing the
raw source data. An example of a mashup is the use of cartographic data from Google Maps to add
location information to real estate data, thereby creating a new and distinct Web service that was not
originally provided by either source” (Wikipedia, 2009).
Fleck
every article in the magazine. It is an almost impossible task but somehow people like
Holovaty have figured out how to do it to web pages via an automated process.
Another example of a screen-scraping site is GovTrack.us. Joshua Tauberer
wanted to create a web site for gathering data about the U.S. Congress and putting
that data together in new ways—creating mashups (Tauberer, 2006). The government
does post the data that he wants online but in “scattered locations” (Tauberer, 2006).
For example, “legislation is posted in one place and votes on the very same
legislation in another” (Tauberer, 2006). Each day GovTrack screen-scrapes the
various locations and then “normalizes” the data and creates XML files (Tauberer,
2006). All the work that he does including the programming are available publically;
he considers it an “independent, non-partisan, non-commercial, and open-source
website” (Tauberer, 2006). The site has been referenced in the footnotes for federal
court cases.
One of the important contributions of screen-scraping hacks is “the fact that
their hacked data is offered” back to the community and to the government site “in a
structured and open format” (Brito, 2007). In essence they are giving us what the
government should be providing in the first place. Third parties can now access this
data and create new applications. MAPLight.org, a site that explores the connection
between money and politics, does exactly that by mashing together congressional
voting information from GovTrack.us with campaign finance information from
OpenSecrets.org (along with additional data sources) (Brito, 2007).
Fleck
To Sell or Not to Sell
Government data has value and can be sold to private firms. This has been a
long-standing practice within government, which has used this argument as a reason
to restrict access.9 But there are good arguments to distribute the data freely and as
widely as possible. Third party companies can still repackage the data and sell it in
innovative ways. The more widespread the data, the more experimentation and the
more government can expect back from the open sharing.
Selling data for profit has been a major issue in the domain of geographic
information systems (GIS) on state and local levels. GIS departments are branding
datasets created for public purposes with intellectual property and ownership rights
and “attempting to generate revenue streams from secondary uses of the data being
made by citizens and businesses” (Onsrud, 1998). This negates the idea of this data as
a public good. The bureaucrats often state that they are protecting the “public trust”
by making sure private firms don’t benefit at the expense of the taxpayer (Onsrud,
1998). But treating the information as a public good and distributing it as widely as
possible serves the same purpose by creating a level playing field where anyone can
use the data as he or she sees fit. The government’s loss of a revenue stream can be
made up by third parties freely sharing innovative tools to mine the data. These tools
will also be shared with government. Governments that support open access to
information “believe that stimulation of the private sector economy in a community
and growth of the tax base through an open government information commons results
in greater economic benefits for the community in the long run than restrictive
9
I discuss some of this later in talking about Carl Malamud’s campaigns to bring the Security and
Exchange Commission’s EDGAR database and the Patent and Trade Office’s patent listings to the
Internet. Both agencies had lucrative deals to sell the information to private companies.
Fleck
government information practices” (Onsrud, 1998).
In looking at three distribution models for GIS data, Lopez (1996) makes a
convincing argument for non-proprietary information policies and free and open
access of GIS data.10 First, it keeps government out of the business of creating value-
added products and services demanded by end-users and thus minimize costs as
agencies simply “…disseminate the raw data sets in more or less the same condition
that they use it internally” (Lopez, 1996). Second is that the incremental cost of
dissemination will go down as state and local government become increasingly
networked (Lopez, 1996). Finally, government agencies should be aware of the tax
contributions from the private sector and economic growth. Releasing GIS data to the
public domain fosters “a favorable environment for emerging information industries”
and this is consistent “with the near-universal goals of encouraging local economic
development in high technology” (Lopez, 1996).
In testimony before the Legislative Counsel Committee of the Oregon State
Legislature, Malamud (2008) looked at the issue of copyrighting and selling public
data. He points to analysis by William Patry, an expert on copyright and the author of
the 7-volume treatise Patry on Copyright, that the law does not support copyrighting
public information although it is still common at the state and local level. Oregon
publishes the Oregon Revised Statutes and sells the license for $30,000. Malamud
argues that Oregon can continue to maintain the revenue stream from selling the
$30,000 version of the statutes but can also distribute the data to the public.
Corporations and law firms will purchase the $30,000 version while students and
10
Besides the open access model, Lopez (1996) looks at cost-recovery scenarios where agencies
charge user fees to public and private parties that want to use the data; and public-private partnerships
where a government agency negotiates an agreement with a single commercial data vendor.
Fleck
small businesses will work with the free data.
In his testimony, Malamud draws an analogy between public information and
infrastructure:
Works of the government are in the public domain because we understand that
business builds on top of these works of government as infrastructure, just as
they build business on top of our other infrastructures such as water, roads, and
electricity. One of the least-recognized but certainly most important roles of
government is as an information provider, and one of the little heralded things
that make our United States different from other countries is the remarkably
effective role our government has played in providing that information
infrastructure.
(Malamud, 2008)
Open Sourcing
The Free and Open Source Software (FOSS) movement has
proven the willingness of people to provide their time to build
compelling software (Weber, 2005). Although public goods theory
shows that nonrival and nonexcludable goods should encourage
free rides, FOSS has shown that talented programmers are willing to
give of their time to build free software (Weber, 2005). This is
already happening with the screen-scraping projects and I will
discuss newer projects in the case studies section. The FOSS
community is building software tools to provide interfaces and build
applications based on openly accessed government data. Projecting
this scenario into the future, it may be possible to standardize on
data formats and application protocols that could then be used by
all governments and from this provide interfaces globally to
research and view data. These applications are then available to the
government thus saving money on internal application
Fleck
development. According to Weber (2005) several U.S. government
departments including the Defense Department, Department of
Energy, and National Security Agency, already work with open
source software.
There is also no reason not to allow commercial vendors to
build custom interfaces to the data and charge for access. If they
are able to design a compelling interface for working with the data
and find customers willing to pay for access to that interface, then
they should be allowed to benefit economically. However they must
not have any data access beyond what the FOSS community is
allowed. The playing field must be kept level.
Companies might also profit in these scenarios by offering
tech support for the FOSS applications. This is a model already in
use in the FOSS community (Weber, 2005). In terms of this
initiative, a company might also provide statistical help in analyzing
the data or even sell their services to nonprofits, political
campaigns, and corporations to build data-mining interfaces.
The next two sections will 1) look at legislation supporting
U.S. government transparency and 2) examine case studies of
Washington D.C.’s DCStats implementation and Carl Malamud’s
work in freeing Securities and Exchange Commission data.
Fleck
Supporting Laws on the Road to Transparency: U.S. Government Transparency

Legislation
This section is a look at selected U.S. legislation supporting open access to
government data.
Paperwork Reduction Act of 1995
President Bill Clinton signed the Paperwork Reduction Act into law on May
22, 1995. Besides reducing paper usage, the act requires the Federal government to
“provide for the dissemination of public information on a timely
basis, on equitable terms, and in a manner that promotes the utility
of the information to the public and makes effective use of
information technology” (U.S. National Archives, 1995). In addition it
directs the government to “ensure the greatest possible public
benefit from and maximize the utility of information created,
collected, maintained, used, shared and disseminated by or for the
Federal Government” (U.S. National Archives, 1995).
The act endorses the idea of sharing government data and its
injunction to “ensure the greatest possible public benefit” supports
open access to data in multiple formats. This is the quickest way to
provide the data and circumvents the need to build web sites that
usually only provide agency- or department-selected subsets of
data. The closer we can go to the source of the info, the more we
can satisfy a greater number of constituents.
The act served as a base for the first major movement of U.S.
Fleck
government data to the web in 1998 when Carl Malamud brought
Security and Exchange Commission’s EDGAR database and Patent
and Trademark Office databases to the Internet (Malamud, 1999). A
later section of the paper will detail his efforts
Electronic Freedom of Information Act Amendments
The 1966 Freedom of Information Act (FOIA) was amended in 1996 under
President Clinton to require that federal agencies make certain types of
records, created after November 1, 1996, available electronically
(U.S. Department of Justice, 1996). It also requires agencies to
provide “reading rooms” where all records can be viewed in
electronic format. By requiring electronic publication of more data,
President Clinton hoped that there would be less need to invoke the
FOIA to obtain information. In his remarks at the signing, he stated:
The legislation I sign today brings FOIA into the information and
electronic age by clarifying that it applies to records maintained
in electronic format. This law also broadens public access to
government information by placing more material on-line and
expanding the role of the agency reading room. As the
Government actively disseminates more information, I hope that
there will be less need to use FOIA to obtain government
information.
(U.S. Department of Justice, 1996)
E-Government Act of 2002
One of the major pieces of legislation drafted to directly address U.S.
Government web presence was the E-Government Act of 2002 signed into law under
President George W. Bush (Library of Congress Thomas, 2002). This act established
the Office of Electronic Government and is charged with removing “information
barriers between federal agencies to give the public easier online access to data and
Fleck
services” (Raney, 2002). The law calls for Federal agencies to provide ways for the
public to comment and defines “electronic Government” as “the use by Government
of web-based Internet applications and other information technologies, combined
with processes that implement these technologies, to: (1) enhance the access to and
delivery of Government information and services; or (2) bring about improvements in
Government operations” (Library of Congress Thomas, 2002).
On one level, this is a step backward from the Paperwork Reduction Act. It
resulted in a more top-down approach, serving the data that the government wants
shared or that government employees have the time to share. It also focuses on
providing web-based services like renewing automobile license tags or paying for
city-delivered utility services like water and trash. These service functions of
government web sites are necessary but should not be confused with engaging
citizens in the processes of governing. It is part of a makeover to rebrand the
government as a business providing services to customers rather than engaging
citizens in the process of democracy.
This attitude is confirmed by Darrell M. West, director of the Taubman Center
for Public Policy at Brown University, as quoted in the Raney (2002) article. He says
that this law pushes the government to a more corporate style and might even include
corporate technology workers working for the government in an exchange program.
On the U.S. GSA site documents discuss this “corporate” style and state that the
Federal government should use the tools developed in the private sector since
corporations have solved many of the problems inherent in creating and maintaining
web sites (Patch, 2003). I see a concern here, as corporate environments are not
Fleck
known for openness and sharing operational data to the public.
The federal Regulations.gov site—“ the public face of the Federal E-
Government eRulemaking Program”—resulted from this legislation
(Regulations.gov, ND). The site “facilitates public participation in the Federal
regulatory process by improving the public's ability to find, view, and comment on
Federal regulatory actions” (Regulations.gov, ND). Groups such as the Congressional
Research Service and the American Bar Association have criticized the poor design
of the site and lack of funding for improvements (Brito, 2007; Lindeman, 2008
10/20). Brito (2007) has gone so far as to say that “While efficient in theory,
consolidation may be a step backward if the centralized database does more to
obscure data than to make it easily accessible.” Noveck (2004) called the initiative
“perhaps the most far-reaching and important such governmental transformation ever
effected” but noted that “this radical overhaul of the administrative process is
conducted in a closed and almost secretive manner without public consultation”
ignoring “how individuals and groups communicate and work together to solve
problems.”
The site had a major redesign in 2008 and added an RSS feed and revised
searching features.11 This also allowed Jerry Brito to build a companion site called
OpenRegulations.org that is an “alternative interface to the federal government's
Regulations.gov regulatory dockets database” (Brito, 2009). OpenRegulations.org
offers features not available at the government site and indicates that third parties will
enhance and improve data retrieval.
The Federal Funding Accountability and Transparency Act of 2006

11
RSS is a structured format (actually XML) and allows for subscribing via various tools.
Fleck
On September 26, 2006, President Bush signed the Federal Funding
Accountability and Transparency Act (FFATA) (S 2590) into law (Hatch, 2006).12
This law requires the Office of Management and Budget to create a new public
database that will share information about entities that receive federal grants, loans,
and contracts (Hatch, 2006). The government awards over one trillion dollars each
year in this type of financial transaction, a figure that represents nearly one-third of
the federal government’s expenditures and obligations (Hatch, 2006).
The web-enabled result of FFATA is USAspending.gov, a site that is
searchable and free to the public. It provides names of entities, amounts of awards,
the entity location, and other information including transaction type and funding
agency (Miller, 2008). It also delivers the data in an accessible format13 via an API14
for pulling the raw data out and repurposing it and/or mixing it with other data to
create a “mashup” (Office of Management and Budget, 2009).
This site goes a long way towards satisfying President Obama’s transparency
initiatives and giving citizens new ways to view government data. The data is still
somewhat filtered via the API15 but it is “raw” enough to make it very useful for
mashups.
12
As previously mentioned, this law was sponsored by then Senator Barack Obama (IL) and Senator
Tom Coburn (OK).
13
The data is delivered in an XML format. See Footnote 5.
14
API stands for “application programmer interface.” An API makes it easier for a programmer to
work with an application or an operating system by providing a documented interface.
15
Robinson (2008) notes that although an API delivers search results “…having to work through an
interface sometimes limits developers from making innovative, unforeseen uses of the data.” He also
notes that the Strengthening Transparency and Accountability in Federal Spending Act of 2008 (S.
3077), mandates use of an API for no good reason. (This bill has not been reintroduced in the current
legislative session as of this writing.)
Fleck
Case Studies in Open Access
Washington, D.C. and DCStats (now known as CityDW)
On June 12, 2006, the City of Washington, D.C., launched its
DCStats open government project. This project carries out principles
at the heart of Obama’s mandates about transparency and
providing the information.16 The district provides city operational
data in a variety of formats to the public. The goal of the DCStats
project is to make “the delivery of taxpayer funded services
transparent and accountable” (Udell, 2006).
Today the DCStats project is known as the Citywide Data
Warehouse (CityDW) and provides 275 datasets from multiple
agencies. It is an excellent model as to how urban areas can give
constituents the capability of tracking government performance.
The City provides the datasets in five different formats. These
include:
1. The structured Extensible Markup Language (XML) which

allows for easy automated reading by computers;
2. A comma-separated text file (CSV) which can be read by
desktop computer software like Excel and networked
software like Google Documents;
3. An Atom data feed for subscribing via an RSS or feed
reader like Google Reader or Microsoft Explorer 7. These
feeds are updated once each hour. It is also possible to
subscribe to this feed from a web site and restructure the
data for public display.
4. An ESRI17 file for use with ESRI-compatible mapping
16
It also adheres to the “Open Government Data Principles” found at
http://resource.org/8_principles.html.
17
“Used for consumption by ESRI-compatible mapping applications. Most datasets in ESRI format are
updated on a monthly or quarterly basis as they are not ‘operational’ in nature”(District of Columbia,
Fleck
applications.
5. A KML file of geospatial data for use with Google Earth,
Google maps, or similar applications.
I chose to review a sample CSV data file from the Department
of Consumer and Regulatory Affairs. The data consisted of building
permit applications for the entire city. I downloaded the file to my
computer and opened it with Microsoft Excel as a spreadsheet. The
spreadsheet (Figure 1) displayed thirty-four columns and 972 rows
spanning permit applications from September 15, 2008 to April 18,
2009. Data attributes, represented by column headings, included
project address, description of the work, permit applicant, project
owner, fees paid, latitude and longitude of the project, zip codes,
and other local geographic identifiers.
Figure 1. Portion of the spreadsheet.
The same information is also available at the CityDW web site
in all the formats previously listed. With the KML data format a link
brings you to a page with the data overlaid on a Google map. Each
permit application becomes a “pin” on the map as shown in Figure
Office of Chief Technology Officer, ND).
Fleck
2.
Figure 2. KML data overlaid on a Google map.
Suzanne Peck was D.C.’s CTO when DCStats launched. In an
interview with Jon Udell (2006), she listed three reasons that the
city wanted to provide this data:
1. If the City is taking in tax dollars to organize efficient citizen,

business, and visitor services, it needs to be accountable “for
the efficiency and level of service with which they use those
dollars” (Udell, 2006). Providing raw data openly over the web
allows anyone to monitor efficiency levels across the city.
2. Data historically has been locked in “silos” and only available

vertically within departments and agencies but not
horizontally across bureaucratic boundaries. From this
standpoint, CityDW allows for better data sharing across city
department boundaries as it also allows anyone to compile
and aggregate the data for comparisons across department
walls.
3. The technology itself had reached the maturity level where
Fleck
these tasks could be manifested.
Vivek Kundra, CTO for the District of Columbia in 2008,
launched a contest in October of that year called “Applications for
Democracy” (District of Columbia, 2008). The contest invited
software developers to design applications for the web or mobile
devices that would make the D.C. data feeds more accessible and
useful for the public (District of Columbia, 2008). The idea was to
create mashup applications (see Footnote 8). An example
application might be similar to the Google map mashup in Fig. 1
that shows building permit sites. Developers were required to use
open source tools or to release their applications as open source
with the underlying programming code publically available.
Developers submitted forty-seven applications for the contest.
The District estimated the value of all the applications at $2 million
and compared the process to the traditional RFP process.
Applications for Democracy took six days to launch and cost very
little. An RFP process might take years and cost over $1 million
(Corbett, 2008).
Top prize winning applications were “DC Historic Tours” and
“iLive.at” (Apps for Democracy, 2008a, 2008b). “DC Historic Tours”
by Adam Boalt at Boalt Interactive combines data feeds from the
Flickr photo site (Flickr.com) and Wikipedia (Wikipedia.org) with a
Google map to create customized tours of the District of Columbia
Fleck
(Apps for Democracy, 2008a). “iLive.at” by Travis Hurant, Tim
Koelkebeck, and Brian Sobel presents you with custom information
for an exact location in the District including recent crimes,
demographic information, and an “errands” category listing
shopping centers and post offices (Apps for Democracy, 2008b).
Other winning applications include crime listings based on your
telephone number; a carpooling system; and a biking guide that
lists bike routes, Metro stations, creates custom maps, and even
maps out bike theft sites. All of these applications utilize D.C.
CityDW data in some way. All were built at no cost to the District.
The various feeds that the District provides made creating the
mashups relatively easy. All can be moved to other servers,
modified, and shared.
Case Study: Carl Malamud
Carl Malamud is perhaps the best known of the open access
pioneers. Through his efforts, the Securities and Exchange
Commission (SEC) Electronic Data Gathering, Analysis and Retrieval
(EDGAR) database was opened to public access in 1998. He was
also instrumental in gaining access to the Patent and Trade Office
databases. Additionally he added access to databases from the
General Services Administration, the Federal Election Commission,
the Federal Reserve Board and the Government Printing Office
(Malamud, 1998). He also provided audio feeds from the floor of the
Fleck
House and Senate (Markoff, 1998b). The work he did in providing
this access was through a nonprofit called the Internet Multicasting
Service, a public research corporation.
The SEC’s EDGAR system is “the world's most important and
valuable source of information on corporate activities, and includes
the full text of a large number of company disclosure reports on
company finances and operations” (Love, 1993). Originally via a
contract with Mead Corporation, the FCC was going to provide
access to the information at costs from $30,000 to $175,000 per
year (Love, 1993). In 1989, library organizations expressed interest
in using the SEC data and gaining access through the 1,400
member federal Depository Library Program (DLP) (Love, 1993).
Mead, the SEC contractor, stated that it would cost $18 million to
provide the access and the effort was defeated (Love, 1993).
In 1991, the Taxpayer Assets Project (TAP), a group founded
by Ralph Nader, began criticizing Mead’s arrangement via
testimony to the federal Joint Committee on Printing (Love, 1993)
and asking for broader public access to the data. Discussions
ensued with the SEC and congressional staff. Letters were
circulated via the Internet in support of the initiative and groups like
the American Economic Association, and the American Library
Association weighed in in favor of the proposal. TAP sent a letter to
Representative Edward Markey (Massachusetts), chair of the
Fleck
Subcommittee on Telecommunications and Finance, and Senator
Herbert Kohl (Wisconsin) asking for broader access.
The SEC refused to allow online access and Mead threatened
a lawsuit if the SEC did provide the access (Love, 1993). The SEC also
claimed that providing the data on the internet was “technically impossible” and even
if it were possible, “the only people interested in SEC fillings were Wall Street
Fatcats and they didn't really need subsidized access to data they were willing to pay
for” (Malamud, 1999).
Rep. Markey spoke with Malamud about the issue in 1993. Malamud was
intrigued: “If something is technically impossible, I get interested,”
(Malamud, 1999).
Rep. Markey’s staff arranged a meeting with Malamud and the SEC to
“discuss the idea of giving us the data and letting us put together an Internet site.
There was a bit of pushback, to say the least” (Malamud, 1999). Malamud continues:
The problem was the 70's era data processing system that the SEC had put in
place in the late 80's. The deal was that EDGAR was way too rough for
consumers to digest. It needed, to speak the MIS lingo of the time, “value-add.”
Who would add value? Well, the SEC had cut a contract with a data wholesaler
who would add value. The wholesaler, in turn, would sell to information
retailers who would add even more value. Then, the information would be sold
on the retail information market to the Wall Street crowd who had an interest in
the data. Obviously, if we gave away all this information on the Internet, it
would subvert our entire Free Enterprise System.
In that meeting with the SEC and the Chairman's staff, my favorite moment was
when we got to the question of why in the world people would want to see
EDGAR data. I maintained that the Internet was full of lots of people—students,
journalists, senior citizen investors—who were dying for access to this data. The
SEC felt that only a few people would want to see EDGAR documents, and
besides the Internet (or “the ARPANET” as they kept referring to it) “didn't
have the right kind of people.”
Now, this was a cheap shot, and I understood that what they meant was “there
weren't a lot of people, just a few researchers,” but I couldn't resist.
“The right kind of people?” I said, rising up in my chair. “I think the American
Fleck
people are the right kind of people.”
So much for the idea of working cooperatively.

(Malamud, 1999)
With a National Science Foundation grant and some borrowed computers
from Sun, Malamud launched a free EDGAR system on the Internet in January 1994.
This effort undercut the retail information industry of the time that was making
several hundred million dollars a year selling the information. He ran the site with
Brad Burdick for eighteen months. At its peak, 50,000 people visited the site each day
(Malamud, 1999).
Malamud’s goal however was to get the SEC to maintain the site. He did not
want to be in the online database business (Malamud, 1999). To that end, Malamud
forced the issue by putting a notice on the site in August 1995 stating that access to
EDGAR would shut down in 60 days and users should contact the SEC for continued
access (Malamud, 1999). Users of the site sent 15,000 messages to SEC Chairman
Arthur Levitt and the Wall Street Journal, New York Times and Associated Press ran
stories (Malamud, 1999; Malamud, 1998). This put pressure on the SEC and they
finally decided it was time to take the service under their roof and support it. By
October 1, Malamud’s cutoff date, the SEC EDGAR site was fully operational. Since
then, they have made substantial enhancements to the site (Malamud, 1999). The
EDGAR site now gets up to 500,000 users each day (Macavinta, 1998).
Malamud had added access to Patents and Trade Office (PTO) data in addition
to SEC data and planned on ending that with the end of SEC publication. He was not
so successful convincing the PTO to maintain the servers. PTO Commissioner Bruce
Lehman felt the fees generated by retailers repackaging the data—estimated at $20
Fleck
million—were necessary for the financial survival of the self-supported office
(Malamud, 1998; Macavinta, 1998). In an April 27, 1998 letter to then Vice President
Al Gore, Malamud argued that open access to the patent database would attract new
users who could never afford the fees and that current customers would still prefer
and pay for the repackaging (Malamud, 1998). Malamud (1998) stressed that opening
up the patent information would give the U.S. a much more robust intellectual
property market: “Markets are based on information and we cannot have an efficient
market for intellectual property on a global basis if we hide the documents that define
that property.”
In June of 1998, Lehman and the PTO changed course and announced that the
over 2 million patent documents would become available via the Internet in a
keyword-searchable database (Markoff, 1998a). Malamud announced that he would
take down his site (Markoff, 1998a).
Malamud continues his open access activism today from the nonprofit site
Public.Resource.Org. The site is a storehouse of government documents available for
download including nonprofit tax returns, state and local building codes and
regulations, and Smithsonian Institution images (Schwartz & Mackey, 2009). Also
available is a growing film archive (915 films as of June 2009) called FedFlix. These
are films originally produced for the federal government (Schwartz & Mackey, 2009).
Malamud is currently working on “freeing” data out of the U.S. Courts Public Access
to Court Electronic Records (PACER) search engine (Singel, 2008) and running for
Public Printer of the United States (Singel, 2009).
Conclusion
Fleck
This is what transparency reform looks like. Complicated, messy, confusing,

often bipartisan, often initially unsuccessful, and helpfully spurred on through
public involvement. (Wonderlich, 2009)
Over the course of writing this paper, open access at the Federal level has
grown exponentially. Data.gov was built to “increase public access to high value,
machine readable datasets generated by the Executive Branch of the Federal
Government” (Federal CIO Council, 2009). Recovery.gov was established to provide
education, transparency, and accountability around the American Recovery and
Reinvestment Act (Recovery Accountability and Transparency Board, 2009). The
implementations are not without controversy. Miller (2009) at the Sunlight
Foundation Blog points out that the Office of Management and Budget’s final
“guidance” does not include providing Recovery Act data in a raw format and calls
this a “Significant failure.” Madrigal (2009) points out that Data.gov had only fifty
data sets available when launched in May.18 But given President Obama’s
Memorandum for Transparency and Open Government (Obama, 2009) and his choice
of D.C.’s Vivek Kundra as CIO, the Federal government is showing that it believes
open access to data that fuels policy decisions will nurture an engaged citizenry and
promote democracy. Now these efforts need to cascade downward to state and local
levels where government entities should emulate the work of Washington, D.C. and
it’s CityDW online data warehouse.
This paper should leave no doubt that open access to raw data is in the best
interests of the public good. It should also prove that third parties (including average
18
From personal experience, I have found problems with the data formats at Data.gov. One dataset that
I downloaded was only available as a Windows executable file and not accessible with the Apple
operating system. The other dataset was in a dBase format. This is an “ancient” database format that is
not easily readable today.
Fleck
citizens) will gather the data for analysis and reuse and that should be seen as a
benefit for the government as it can share in the process. This truly moves us closer to
a participatory democracy.
Fleck
Bibliography
Apps for Democracy. (2008a). DC Historic Tours | Apps for Democracy. Retrieved
June 3, 2009, from http://www.appsfordemocracy.org/dc-historic-tours/.
Apps for Democracy. (2008b). iLive.at | Apps for Democracy. Retrieved June 3,
2009, from http://www.appsfordemocracy.org/iliveat/.
Brito, J. (2007). Hack, Mash, & Peer: Crowdsourcing Government Transparency.

Hack, Mash & Peer: Crowdsourcing Government Transparency. Retrieved
December 8, 2008, from http://papers.ssrn.com/sol3/papers.cfm?
abstract_id=1023485.
Brito, J. (2009). OpenRegulations.org. Retrieved June 14, 2009, from

http://www.openregulations.org/.
Clift, S. (2007). Ten Practical Online Steps for Government Support of Democracy.
USA Services Intergovernmental Newsletter, How E-Government is Changing
Society and Strengthening Democracy, (20).
Corbett, P. (2008). Government 2.0: The Rise of Citizen Innovation Through Open
Data | iStrategyLabs. Retrieved June 2, 2009, from
http://www.istrategylabs.com/government-20-the-rise-of-citizen-innovation-
through-open-data/.
District of Columbia. (2008). Mayor Fenty Announces Winners of Applications for

Democracy Contest - Releases - Office of the Chief Technology Officer.
Retrieved June 2, 2009, from
http://newsroom.dc.gov/show.aspx/agency/octo/section/2/release/15427.
Federal CIO Council. (2009). Data.gov. Retrieved July 3, 2009, from

http://www.data.gov/.
Hatch, G. L. (2006). The Federal Funding Accountability and Transparency Act:

Background, Overview, and Implementation Issues.
Holovaty, A. (2005). Announcing chicagocrime.org | Holovaty.com. Retrieved June

21, 2009, from http://www.holovaty.com/writing/chicagocrime.org-launch/.
Jardin, X. (2008). Change.gov - Boing Boing. Change.gov - Boing Boing. Retrieved

January 21, 2009, from http://boingboing.net/2008/11/06/changegov.html.
Library of Congress THOMAS search engine. (n.d.). E-Government Search Results -

THOMAS (Library of Congress). Retrieved December 8, 2008, from
http://thomas.loc.gov/cgi-bin/bdquery/z?
d107:HR02458:@@@L&summ2=m&%7CTOM:/bss/d107query.html
Fleck
%7C#summ.
Lindeman, R. (2008). ABA Report on E-Rulemaking Proposes Overhaul of

Government-Wide Program. Daily Report for Executives.
Lopez, X. R. (1996). Stimulating GIS innovation through the dissemination of

geographic information. URISA-WASHINGTON DC-, 8, 24-37.
Love, J. (1993). SEC'S EDGAR on Net, What Happened and Why. Retrieved June 5,
2009, from http://w2.eff.org/Activism/edgar_grant.announce.
Macavinta, Courtney. (1998). Patent office slammed for not posting data - CNET
News. Retrieved June 6, 2009, from http://news.cnet.com/Patent-office-
slammed-for-not-posting-data/2100-1023_3-210894.html.
Madrigal, A. (2009). Data.gov Launches to Mixed Reviews | Wired Science |

Wired.com. Retrieved July 3, 2009, from
http://www.wired.com/wiredscience/2009/05/datagov-launches-to-mixed-
reviews/.
Malamud, C. (1998). Patent and Trademark Databases Should Be On-Line. Retrieved

June 5, 2009, from http://public.resource.org/letter.html.
Malamud, C. (1999). Mappa.Mundi Magazine - The Importance of Being EDGAR.

http://mappa.mundi.net/cartography/EDGAR/index.html.
Malamud, C. (2008). Prepared Statement of Carl Malamud. Retrieved June 6, 2009,

from http://www.scribd.com/doc/3426062/Prepared-Statement-of-Carl-
Malamud?autodown=pdf.
Markoff, J. (, 1998b). Internet Gadfly Wants U.S. To Put More Data Online.
http://www.nytimes.com/library/tech/98/05/biztech/articles/04database.html#
1.
Melber, A. (n.d.). Obama for America 2.0? Obama for America 2.0? Retrieved
January 26, 2009, from http://www.thenation.com/doc/20090112/melber?
rel=hp_currently.
Miller, E. (2008). Sunlight Foundation » USASpending.gov 2.0. Retrieved January

27, 2009, from
http://blog.sunlightfoundation.com/2008/06/03/usaspendinggov-20/.
Miller, E. (2009). No Raw Data on Recovery.gov. Significant Failure — Sunlight

Foundation Blog. Retrieved July 3, 2009, from
Fleck
http://blog.sunlightfoundation.com/2009/06/25/no-raw-data-on-recoverygov-
significant-failure/.
Noveck, B. (2004). The Electronic Revolution in Rulemaking. Retrieved November

16, 2008, from http://papers.ssrn.com/sol3/papers.cfm?abstract-id=506662
O'Brien, T. M. (2009). Vivek Kundra: Federal CIO in His Own Words - O'Reilly
Radar. Retrieved June 11, 2009, from http://radar.oreilly.com/2009/03/vivek-
kundra-federal-cio-in-hi.html.
Obama for America. (ND). Barack Obama: Connecting and Empowering All
Americans Through Technology and Innovation. Retrieved January 20, 2009,
from
http://cairns.typepad.com/blog/files/fact_sheet_innovation_and_technology_pl
an_final.pdf.
Obama, B. (2009). The White House - Press Office - Transparency and Open
Government. Retrieved July 3, 2009, from
http://www.whitehouse.gov/the_press_office/Transparency_and_Open_Gover
nment/.
Office of Management and Budget. (2009). Welcome to USAspending.gov. Retrieved

June 7, 2009, from http://www.usaspending.gov/index.php.
Onsrud, H. J. (1998). Tragedy of the Information Commons. Policy Issues in Modern

Cartography, 141-158.
Raney, F.R. (2002). TECHNOLOGY; New Economy In the next year, the federal
government will move to give the public easier online access to data and
services. . New York Times (1857-Current file), C4. Retrieved December 7,
2008, from http://proquest.umi.com.floyd.lib.umn.edu/pqdweb?
did=731021792&Fmt=7&clientId=2256&RQT=309&VName=HNP.
Recovery Accountability and Transparency Board. (2009). Our Mission |

Recovery.gov. Retrieved July 3, 2009, from http://www.recovery.gov/?
q=content/our-mission.
Regulations.gov. (ND). Regulations.gov. Retrieved June 8, 2009, from

http://www.regulations.gov/search/index.jsp.
Robinson, D., Yu, H., Zeller, W. P., & Felten, E. W. (2008). Government Data and
the Invisible Hand. Yale Journal of Law & Technology, 11. Retrieved from
http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1138083.
Schwartz, J., & Mackey, R. (2009). Steal These Federal Records — Okay, Not
Literally - The Lede Blog - NYTimes.com. Retrieved June 6, 2009, from
Fleck
http://thelede.blogs.nytimes.com/2009/02/13/steal-these-federal-records-okay-
not-literally/?scp=4&sq=%22carl%20malamud%22&st=cse.
Singel, R. (2008). Online Rebel Publishes Millions of Dollars in U.S. Court Records
for Free. Retrieved July 3, 2009, from
http://www.wired.com/politics/onlinerights/news/2008/12/open_pacer?
currentPage=all.
Singel, R. (2009). Rogue Archivist Campaigns to Be Obama’s Printer | Threat Level |

Wired.com. Retrieved July 3, 2009, from
http://www.wired.com/threatlevel/2009/02/rogue-archivist/.
Tauberer, J. (2006). XML.com: GovTrack.us, Public Data, and the Semantic Web.
http://www.xml.com/pub/a/2006/02/08/govtrack-us-public-data-semantic-
web.html?page=1.
U.S. Department of Justice. (1996). FOIA Update: The Freedom of Information Act,
5 U.S.C. sect. 552, As Amended By Public Law No. 104-231, 110 Stat. 3048.
http://www.usdoj.gov/oip/foia_updates/Vol_XVII_4/page2.htm.
U.S. National Archives. (1995). Paperwork Reduction Act (44 U.S.C. 3501.
Retrieved June 5, 2009, from http://www.archives.gov/federal-
register/laws/paperwork-reduction/.
Udell, J. (2006). Jon Udell, Dan Thomas, Suzanne Peck - A conversation with Dan
Thomas and Suzanne Peck about open government: Free MP3 Download.
Retrieved April 30, 2009, from http://beemp3.com/download.php?
file=3288516&song=A+conversation+with+Dan+Thomas+and+Suzanne+Pec
k+about+open+government.
Weber, S. (2005). The Success of Open Source (p. 320). Harvard University Press.
White House Office of the Press Secretary. (2009). The White House - Press Office -
President Obama Names Vivek Kundra Chief Information Officer. Retrieved
June 21, 2009, from http://www.whitehouse.gov/the_press_office/President-
Obama-Names-Vivek-Kundra-Chief-Information-Officer/.
Wonderlich, J. (2009). Senate Reverses Policy, Posts Votes in XML — Sunlight

Foundation Blog. Retrieved June 21, 2009, from
http://blog.sunlightfoundation.com/2009/05/05/senate-reverses-policy-posts-
votes-in-xml/.
Fleck

Fleck Open Access Gov

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Fleck Open Access Gov

Uploaded by

Copyright:

Available Formats

This is a draft version. It will change and may be removed from ScribD.

Please do not cite without permission.

Opening Democracy: Sharing the Raw Ingredients of Policy

If the next Presidential administration really wants to embrace the potential of

MyBO portal (My.BarackObama.com) where social networking merged with

Internet to aggregate information and categorize it for storage in a database. They

conversation and the debate.3

relationship to government. Before his election to the presidency—as Senator from

Funding Accountability and Transparency Act of 2006. On January 21, 2009—the

day of his inauguration—he issued the Transparency and Open Government

Memorandum, which states that his Administration “is committed to creating an

unprecedented level of openness in Government” and to disclosing “information

and connected democracy” to “integrate citizens into the actual business of

government” by “making government data available online in universally accessible

phone contacts (McConnell, 2008).

functions—holding government officials accountable in terms of making sure they

ability to run an open, transparent, participatory, and collaborative government”

information was a difficult and expensive task involving photocopying documents

“release public information online in a structured, open, and searchable manner”

the actual business of democracy and to promote government transparency. Brito

(2007) defines a “structured format” as “information… presented in a format that

guarantee transparency, this open access to government data saves money in

engagement with our democracy. I include a short history of government

transparency legislation and I detail several groundbreaking projects in the area of

screens and republishing data.

formulation based on “pushing” selected information to “consumers.” Federal CIO

“‘Web 2.0’ interactivity built on top of government data by those outside of

unpredictable ways and “highlight patterns that would be otherwise imperceptible in

the source data.”

As proof of the “outside of government” efforts of citizen open-access

“screen-scraping.” “Screen-scraping” involves writing a program that will read in a

One of the classic screen-scraping site using public data is ChicagoCrime.org

created by Adrian Holovaty in May of 2005.7 Holovaty (2005) describes it:

The site is a freely browsable database of crimes reported in Chicago. My scripts

Another example of a screen-scraping site is GovTrack.us. Joshua Tauberer

legislation in another” (Tauberer, 2006). Each day GovTrack screen-scrapes the

he considers it an “independent, non-partisan, non-commercial, and open-source

One of the important contributions of screen-scraping hacks is “the fact that

voting information from GovTrack.us with campaign finance information from

OpenSecrets.org (along with additional data sources) (Brito, 2007).

To Sell or Not to Sell

more government can expect back from the open sharing.

information “believe that stimulation of the private sector economy in a community

government information practices” (Onsrud, 1998).

dissemination will go down as state and local government become increasingly

public domain fosters “a favorable environment for emerging information industries”

development in high technology” (Lopez, 1996).

In testimony before the Legislative Counsel Committee of the Oregon State

small businesses will work with the free data.

In his testimony, Malamud draws an analogy between public information and

The Free and Open Source Software (FOSS) movement has

proven the willingness of people to provide their time to build

compelling software (Weber, 2005). Although public goods theory

shows that nonrival and nonexcludable goods should encourage

give of their time to build free software (Weber, 2005). This is

already happening with the screen-scraping projects and I will

discuss newer projects in the case studies section. The FOSS

community is building software tools to provide interfaces and build

applications based on openly accessed government data. Projecting

this scenario into the future, it may be possible to standardize on

data formats and application protocols that could then be used by

all governments and from this provide interfaces globally to

government thus saving money on internal application

development. According to Weber (2005) several U.S. government