Professional Documents
Culture Documents
Government Datasets
Mohammed Firdaus, Muhd Sharuzzamal Bakri Open Source Tools for Creating Mashups with Government Datas
Introduction About the Speakers
Mohammed Firdaus, Muhd Sharuzzamal Bakri Open Source Tools for Creating Mashups with Government Datas
Introduction What are Mashups?
Mashups
Mohammed Firdaus, Muhd Sharuzzamal Bakri Open Source Tools for Creating Mashups with Government Datas
Challenges Data Sets are Not Available in Machine Readable Form
Mohammed Firdaus, Muhd Sharuzzamal Bakri Open Source Tools for Creating Mashups with Government Datas
Challenges No Data Dictionaries
No Data Dictionaries
Since the data sets that are available were meant for humans
to consume rather machines they are usually published
without any type of data dictionary.
This means that an application developer will have to make
assumptions about the structure of each field e.g. whether it’s
unique, whether it’s a multi-value field, which fields are
mandatory/option.
These assumptions may or may not turn out be correct as you
see more and more data in the data set.
Mohammed Firdaus, Muhd Sharuzzamal Bakri Open Source Tools for Creating Mashups with Government Datas
Challenges New Data Sets Constantly Become Available
Mohammed Firdaus, Muhd Sharuzzamal Bakri Open Source Tools for Creating Mashups with Government Datas
Challenges Lack of Standards Across Agencies
Mohammed Firdaus, Muhd Sharuzzamal Bakri Open Source Tools for Creating Mashups with Government Datas
Challenges Summary
In Summary
Mohammed Firdaus, Muhd Sharuzzamal Bakri Open Source Tools for Creating Mashups with Government Datas
Graphs Introduction to Graphs
What is a Graph?
Mohammed Firdaus, Muhd Sharuzzamal Bakri Open Source Tools for Creating Mashups with Government Datas
Graphs Types of Graphs
Types of Graphs
Mohammed Firdaus, Muhd Sharuzzamal Bakri Open Source Tools for Creating Mashups with Government Datas
Graphs Types of Graphs
Mohammed Firdaus, Muhd Sharuzzamal Bakri Open Source Tools for Creating Mashups with Government Datas
Graphs Types of Graphs
Mohammed Firdaus, Muhd Sharuzzamal Bakri Open Source Tools for Creating Mashups with Government Datas
Graphs Types of Graphs
Mohammed Firdaus, Muhd Sharuzzamal Bakri Open Source Tools for Creating Mashups with Government Datas
Graphs Types of Graphs
Mohammed Firdaus, Muhd Sharuzzamal Bakri Open Source Tools for Creating Mashups with Government Datas
Graphs Types of Graphs
Mohammed Firdaus, Muhd Sharuzzamal Bakri Open Source Tools for Creating Mashups with Government Datas
Graphs Types of Graphs
Source: http://www.flickr.com/photos/greenem/11696663/
http://en.wikipedia.org/wiki/File:WorldWideWebAroundWikipedia.png
Property Graphs
Mohammed Firdaus, Muhd Sharuzzamal Bakri Open Source Tools for Creating Mashups with Government Datas
Graphs Property Graphs
Property Graphs
Source: http://wiki.github.com/tinkerpop/gremlin/defining-a-property-graph
Mohammed Firdaus, Muhd Sharuzzamal Bakri Open Source Tools for Creating Mashups with Government Datas
Data Sets Treasury Procurement Data
Source: http://myprocurement.treasury.gov.my/index.php/en/list-keputusan-tender
Mohammed Firdaus, Muhd Sharuzzamal Bakri Open Source Tools for Creating Mashups with Government Datas
Data Sets Treasury Procurement Data
Fields
Mohammed Firdaus, Muhd Sharuzzamal Bakri Open Source Tools for Creating Mashups with Government Datas
Data Sets Treasury Procurement Data
Mohammed Firdaus, Muhd Sharuzzamal Bakri Open Source Tools for Creating Mashups with Government Datas
Data Sets Treasury Procurement Data
The Dump
Mohammed Firdaus, Muhd Sharuzzamal Bakri Open Source Tools for Creating Mashups with Government Datas
Data Sets Issues with this Data Sets
Missing Fields
Mohammed Firdaus, Muhd Sharuzzamal Bakri Open Source Tools for Creating Mashups with Government Datas
Data Sets Issues with this Data Sets
Mohammed Firdaus, Muhd Sharuzzamal Bakri Open Source Tools for Creating Mashups with Government Datas
Data Sets Issues with this Data Sets
Mohammed Firdaus, Muhd Sharuzzamal Bakri Open Source Tools for Creating Mashups with Government Datas
Data Sets Issues with this Data Sets
Mohammed Firdaus, Muhd Sharuzzamal Bakri Open Source Tools for Creating Mashups with Government Datas
Data Sets Modeling
Mohammed Firdaus, Muhd Sharuzzamal Bakri Open Source Tools for Creating Mashups with Government Datas
Data Sets Modeling
Example
Mohammed Firdaus, Muhd Sharuzzamal Bakri Open Source Tools for Creating Mashups with Government Datas
Graph Databases and Neo4j Neo4j - Introduction
Neo4j
Mohammed Firdaus, Muhd Sharuzzamal Bakri Open Source Tools for Creating Mashups with Government Datas
Graph Databases and Neo4j Neo4j - Introduction
Neo4j
Written in Java.
Bindings available for Python, Ruby, Clojure, Erlang, Groovy,
Scalan and PHP.
We will be using the Python bindings in this talk.
An embedded database, meaning that it runs in the same
process space as the application.
There’s a standalone REST server for those who prefer it.
Mohammed Firdaus, Muhd Sharuzzamal Bakri Open Source Tools for Creating Mashups with Government Datas
Graph Databases and Neo4j Inserting into Neo4j
import neo4j
db = neo4j.GraphDatabase("db")
Mohammed Firdaus, Muhd Sharuzzamal Bakri Open Source Tools for Creating Mashups with Government Datas
Graph Databases and Neo4j Inserting into Neo4j
Mohammed Firdaus, Muhd Sharuzzamal Bakri Open Source Tools for Creating Mashups with Government Datas
Graph Databases and Neo4j Inserting into Neo4j
Mohammed Firdaus, Muhd Sharuzzamal Bakri Open Source Tools for Creating Mashups with Government Datas
Graph Databases and Neo4j Inserting into Neo4j
Indexing Nodes
Mohammed Firdaus, Muhd Sharuzzamal Bakri Open Source Tools for Creating Mashups with Government Datas
Graph Databases and Neo4j Inserting into Neo4j
The Result
Mohammed Firdaus, Muhd Sharuzzamal Bakri Open Source Tools for Creating Mashups with Government Datas
Graph Traversals
Mohammed Firdaus, Muhd Sharuzzamal Bakri Open Source Tools for Creating Mashups with Government Datas
Graph Traversals
Mohammed Firdaus, Muhd Sharuzzamal Bakri Open Source Tools for Creating Mashups with Government Datas
Graph Traversals
Problem
Lets use graph traversal to find all the companies who have been
awarded contracts by Kementerian Kesihatan.
Mohammed Firdaus, Muhd Sharuzzamal Bakri Open Source Tools for Creating Mashups with Government Datas
Graph Traversals
Mohammed Firdaus, Muhd Sharuzzamal Bakri Open Source Tools for Creating Mashups with Government Datas
Graph Traversals Traversal Framework
Mohammed Firdaus, Muhd Sharuzzamal Bakri Open Source Tools for Creating Mashups with Government Datas
Graph Traversals Traversal Framework
with db.transaction:
moh = ministries["KEMENTERIAN KESIHATAN"]
contractors = Contractors(moh)
for c in contractors:
print c["name"]
Mohammed Firdaus, Muhd Sharuzzamal Bakri Open Source Tools for Creating Mashups with Government Datas
Graph Traversals Traversal Framework
Output
Mohammed Firdaus, Muhd Sharuzzamal Bakri Open Source Tools for Creating Mashups with Government Datas
Graph Traversals Traversing Graphs with Gremlin
Gremlin
Mohammed Firdaus, Muhd Sharuzzamal Bakri Open Source Tools for Creating Mashups with Government Datas
Graph Traversals Traversing Graphs with Gremlin
Explanation
Mohammed Firdaus, Muhd Sharuzzamal Bakri Open Source Tools for Creating Mashups with Government Datas
Graph Traversals Traversing Graphs with Gremlin
Explanation
Mohammed Firdaus, Muhd Sharuzzamal Bakri Open Source Tools for Creating Mashups with Government Datas
Graph Traversals Traversing Graphs with Gremlin
Explanation
Mohammed Firdaus, Muhd Sharuzzamal Bakri Open Source Tools for Creating Mashups with Government Datas
Graph Visualizations Gephi
Gephi
Mohammed Firdaus, Muhd Sharuzzamal Bakri Open Source Tools for Creating Mashups with Government Datas
Graph Visualizations Gephi
Mohammed Firdaus, Muhd Sharuzzamal Bakri Open Source Tools for Creating Mashups with Government Datas
Graph Visualizations Gephi
Mohammed Firdaus, Muhd Sharuzzamal Bakri Open Source Tools for Creating Mashups with Government Datas
Mashing Up Adding External Data Sources
Mashing Up
Mohammed Firdaus, Muhd Sharuzzamal Bakri Open Source Tools for Creating Mashups with Government Datas
Mashing Up Adding External Data Sources
Mohammed Firdaus, Muhd Sharuzzamal Bakri Open Source Tools for Creating Mashups with Government Datas
Mashing Up Adding External Data Sources
Mohammed Firdaus, Muhd Sharuzzamal Bakri Open Source Tools for Creating Mashups with Government Datas
Mashing Up Adding External Data Sources
Mohammed Firdaus, Muhd Sharuzzamal Bakri Open Source Tools for Creating Mashups with Government Datas
Mashing Up Traversing to Find Direct/Indirect Awards
The Traverser
class AllTendersDirectIndirect(neo4j.Traversal):
types = [neo4j.Incoming.awarded to,
neo4j.Outgoing.shareholder in]
Mohammed Firdaus, Muhd Sharuzzamal Bakri Open Source Tools for Creating Mashups with Government Datas
Mashing Up Traversing to Find Direct/Indirect Awards
Output
30/2009
35/2009
8/2009
162/2009
JASA/OP/1/2009
Mohammed Firdaus, Muhd Sharuzzamal Bakri Open Source Tools for Creating Mashups with Government Datas
Wrapup Making this Easier
Mohammed Firdaus, Muhd Sharuzzamal Bakri Open Source Tools for Creating Mashups with Government Datas
Wrapup Making this Easier
Mohammed Firdaus, Muhd Sharuzzamal Bakri Open Source Tools for Creating Mashups with Government Datas
Wrapup Making this Easier
Mohammed Firdaus, Muhd Sharuzzamal Bakri Open Source Tools for Creating Mashups with Government Datas