P. 1
Golconde pgCon 2009 Presentation

Golconde pgCon 2009 Presentation

|Views: 176|Likes:
Published by Gavin M. Roy
Golconde is a queue based replication solution for PostgreSQL written in Python. It is designed to be loosely coupled and rely upon existing enterprise messaging systems that have STOMP protocol support. Designed to scale easily and with multi-data center implementations in mind, the application and message queues for distribution live outside of the database. By decoupling Golconde from PostgreSQL it is differentiated from existing replication solutions, moving the workload from the database tier, where CPU, RAM and IO overhead can be very expensive, to a commodity layer where the operational cost for performing the data distribution work is much less expensive.
Golconde is a queue based replication solution for PostgreSQL written in Python. It is designed to be loosely coupled and rely upon existing enterprise messaging systems that have STOMP protocol support. Designed to scale easily and with multi-data center implementations in mind, the application and message queues for distribution live outside of the database. By decoupling Golconde from PostgreSQL it is differentiated from existing replication solutions, moving the workload from the database tier, where CPU, RAM and IO overhead can be very expensive, to a commodity layer where the operational cost for performing the data distribution work is much less expensive.

More info:

Published by: Gavin M. Roy on May 23, 2009
Copyright:Attribution Non-commercial

Availability:

Read on Scribd mobile: iPhone, iPad and Android.
download as PDF, TXT or read online from Scribd
See more
See less

06/20/2010

pdf

text

original

Friday, May 22, 2009

GOLCONDE
René Magritte, 1953

Friday, May 22, 2009

GOLCONDE

* UNLESS YOU START FROM A CLEAN SLATE

Friday, May 22, 2009

GOLCONDE

NOT REPLICATION*

* UNLESS YOU START FROM A CLEAN SLATE

Friday, May 22, 2009

GOLCONDE

NOT REPLICATION* DATA DISTRIBUTION SYSTEM

* UNLESS YOU START FROM A CLEAN SLATE

Friday, May 22, 2009

GOLCONDE

NOT REPLICATION* DATA DISTRIBUTION SYSTEM ONE CANONICAL SOURCE

* UNLESS YOU START FROM A CLEAN SLATE

Friday, May 22, 2009

GOLCONDE

NOT REPLICATION* DATA DISTRIBUTION SYSTEM ONE CANONICAL SOURCE ANY NUMBER OF TARGET RELATIONS

* UNLESS YOU START FROM A CLEAN SLATE

Friday, May 22, 2009

GOLCONDE

NOT REPLICATION* DATA DISTRIBUTION SYSTEM ONE CANONICAL SOURCE ANY NUMBER OF TARGET RELATIONS QUEUE BASED
* UNLESS YOU START FROM A CLEAN SLATE

Friday, May 22, 2009

FEATURES

Friday, May 22, 2009

FEATURES

LIGHT-WEIGHT DAEMON

Friday, May 22, 2009

FEATURES

LIGHT-WEIGHT DAEMON “AUTOSQL” AND CUSTOM HANDLERS

Friday, May 22, 2009

FEATURES

LIGHT-WEIGHT DAEMON “AUTOSQL” AND CUSTOM HANDLERS TWO METHODS OF OPERATION

Friday, May 22, 2009

FEATURES

LIGHT-WEIGHT DAEMON “AUTOSQL” AND CUSTOM HANDLERS TWO METHODS OF OPERATION REAL-TIME VISUALIZATION

Friday, May 22, 2009

WHY GOLCONDE?

Friday, May 22, 2009

WHY GOLCONDE?
GOOD REPLICATION TOOLS EXIST SLONY, LONGDISTE, BUCARDO, ETC

Friday, May 22, 2009

WHY GOLCONDE?
GOOD REPLICATION TOOLS EXIST SLONY, LONGDISTE, BUCARDO, ETC NO “GOLDEN HAMMER”

Friday, May 22, 2009

WHY GOLCONDE?
GOOD REPLICATION TOOLS EXIST SLONY, LONGDISTE, BUCARDO, ETC NO “GOLDEN HAMMER” CURRENT TOOLS ARE RESOURCE INTENSIVE LIVE ON THE DATABASE TIER LIVE IN THE DATABASE

Friday, May 22, 2009

WHY GOLCONDE?

Friday, May 22, 2009

WHY GOLCONDE?
NEEDED A SCALE OUT SOLUTION

Friday, May 22, 2009

WHY GOLCONDE?
NEEDED A SCALE OUT SOLUTION CROSS DATA-CENTER IMPLEMENTATIONS

Friday, May 22, 2009

WHY GOLCONDE?
NEEDED A SCALE OUT SOLUTION CROSS DATA-CENTER IMPLEMENTATIONS MESSAGE BROKERS ARE MORE EFFICIENT FOR QUEUES

Friday, May 22, 2009

WHY GOLCONDE?
NEEDED A SCALE OUT SOLUTION CROSS DATA-CENTER IMPLEMENTATIONS MESSAGE BROKERS ARE MORE EFFICIENT FOR QUEUES DATABASE HARDWARE IS EXPENSIVE ($$$)

Friday, May 22, 2009

WHY GOLCONDE?
NEEDED A SCALE OUT SOLUTION CROSS DATA-CENTER IMPLEMENTATIONS MESSAGE BROKERS ARE MORE EFFICIENT FOR QUEUES DATABASE HARDWARE IS EXPENSIVE ($$$) QUEUE AND CONSUMER GRADE HARDWARE ARE NOT

Friday, May 22, 2009

WHY GOLCONDE?
NEEDED A SCALE OUT SOLUTION CROSS DATA-CENTER IMPLEMENTATIONS MESSAGE BROKERS ARE MORE EFFICIENT FOR QUEUES DATABASE HARDWARE IS EXPENSIVE ($$$) QUEUE AND CONSUMER GRADE HARDWARE ARE NOT NEED CUSTOMIZABLE WORKFLOW FOR DISPARATE RELATIONS

Friday, May 22, 2009

REQUIREMENTS

Friday, May 22, 2009

REQUIREMENTS
STOMP SUPPORTING MESSAGE BROKER

Friday, May 22, 2009

REQUIREMENTS
STOMP SUPPORTING MESSAGE BROKER POSTGRESQL 8.3* OR HIGHER

Friday, May 22, 2009

REQUIREMENTS
STOMP SUPPORTING MESSAGE BROKER POSTGRESQL 8.3* OR HIGHER PYTHON 2.6 PYYAML PSYCOPG2 STOMP.PY

Friday, May 22, 2009

HTTP://ACTIVEMQ.APACHE.ORG/

Friday, May 22, 2009

STOMP

STREAMING TEXT ORIENTED MESSAGE PROTOCOL INTEROPERABLE WIRE FORMAT SUPPORTED BY MULTIPLE BROKERS

HTTP://STOMP.CODEHAUS.ORG/

Friday, May 22, 2009

AMQP

ADVANCED MESSAGE QUEUING PROTOCOL OPEN STANDARD WIRE PROTOCOL EMERGING STANDARD SUPPORTED BY MULTIPLE BROKERS

HTTP://WWW.AMQP.ORG/

Friday, May 22, 2009

OpenAMQP QPID RedHat Enterprise MRG

HTTP://WWW.RABBITMQ.COM/

Friday, May 22, 2009

ARCHITECTURE

Friday, May 22, 2009

ARCHITECTURE

START OF ACTION LIFECYCLE CLIENT-BASED: ENQUEUE FROM APPLICATION TRIGGER-BASED: ENQUEUE FROM POSTGRESQL

Friday, May 22, 2009

ARCHITECTURE

START OF ACTION LIFECYCLE CLIENT-BASED: ENQUEUE FROM APPLICATION TRIGGER-BASED: ENQUEUE FROM POSTGRESQL MULTI-THREADED PYTHON DAEMON

Friday, May 22, 2009

LEXICON

Friday, May 22, 2009

LEXICON

DESTINATIONS ARE THE CANONICAL RELATIONS

Friday, May 22, 2009

LEXICON

DESTINATIONS ARE THE CANONICAL RELATIONS ARE ACTED UPON IN CLIENT-BASED WORKFLOWS

Friday, May 22, 2009

LEXICON

DESTINATIONS ARE THE CANONICAL RELATIONS ARE ACTED UPON IN CLIENT-BASED WORKFLOWS ENQUEUES DATA IN TRIGGER-BASED WORKFLOWS

Friday, May 22, 2009

LEXICON

DESTINATIONS ARE THE CANONICAL RELATIONS ARE ACTED UPON IN CLIENT-BASED WORKFLOWS ENQUEUES DATA IN TRIGGER-BASED WORKFLOWS TARGETS ARE THE DISTRIBUTED RELATIONS

Friday, May 22, 2009

ARCHITECTURE

Friday, May 22, 2009

ARCHITECTURE

DATA IS PASSED IN JSON ENCODED PACKETS

Friday, May 22, 2009

ARCHITECTURE

DATA IS PASSED IN JSON ENCODED PACKETS COMMANDS: ADD (INSERT)

Friday, May 22, 2009

ARCHITECTURE

DATA IS PASSED IN JSON ENCODED PACKETS COMMANDS: ADD (INSERT) DELETE

Friday, May 22, 2009

ARCHITECTURE

DATA IS PASSED IN JSON ENCODED PACKETS COMMANDS: ADD (INSERT) DELETE SET (UPSERT)

Friday, May 22, 2009

ARCHITECTURE

DATA IS PASSED IN JSON ENCODED PACKETS COMMANDS: ADD (INSERT) DELETE SET (UPSERT) UPDATE

Friday, May 22, 2009

MESSAGE EXAMPLES

Friday, May 22, 2009

MESSAGE EXAMPLES

{"ACTION": "ADD", "DATA": {"FRIEND_ID": 47, "USER_ID": 16, "STATUS_ID": 1}}

Friday, May 22, 2009

MESSAGE EXAMPLES

{"ACTION": "ADD", "DATA": {"FRIEND_ID": 47, "USER_ID": 16, "STATUS_ID": 1}} {"ACTION": "DELETE", "RESTRICTION": {"FRIEND_ID": 11, "USER_ID": 126}}

Friday, May 22, 2009

MESSAGE EXAMPLES

{"ACTION": "ADD", "DATA": {"FRIEND_ID": 47, "USER_ID": 16, "STATUS_ID": 1}} {"ACTION": "DELETE", "RESTRICTION": {"FRIEND_ID": 11, "USER_ID": 126}} {"ACTION": "SET", "DATA": {"FRIEND_ID": 112, "USER_ID": 111, "TIMESTAMP": "WED APR 1 13:56:54 2009", "STATUS_ID": 1}}

Friday, May 22, 2009

MESSAGE EXAMPLES

{"ACTION": "ADD", "DATA": {"FRIEND_ID": 47, "USER_ID": 16, "STATUS_ID": 1}} {"ACTION": "DELETE", "RESTRICTION": {"FRIEND_ID": 11, "USER_ID": 126}} {"ACTION": "SET", "DATA": {"FRIEND_ID": 112, "USER_ID": 111, "TIMESTAMP": "WED APR 1 13:56:54 2009", "STATUS_ID": 1}} {"ACTION": "UPDATE", "RESTRICTION": {"FRIEND_ID": 5, "USER_ID": 41}, "DATA": {"TIMESTAMP": "WED APR 1 13:56:38 2009", "STATUS_ID": 3}}

Friday, May 22, 2009

AUTOSQL

Friday, May 22, 2009

AUTOSQL

DEFAULT HANDLER TYPE FOR GOLCONDE

Friday, May 22, 2009

AUTOSQL

DEFAULT HANDLER TYPE FOR GOLCONDE EXAMINES PG_CATALOG DATA FOR SCHEMA

Friday, May 22, 2009

AUTOSQL

DEFAULT HANDLER TYPE FOR GOLCONDE EXAMINES PG_CATALOG DATA FOR SCHEMA CACHES SCHEMA

Friday, May 22, 2009

AUTOSQL

DEFAULT HANDLER TYPE FOR GOLCONDE EXAMINES PG_CATALOG DATA FOR SCHEMA CACHES SCHEMA GENERATES SQL

Friday, May 22, 2009

AUTOSQL

DEFAULT HANDLER TYPE FOR GOLCONDE EXAMINES PG_CATALOG DATA FOR SCHEMA CACHES SCHEMA GENERATES SQL REQUIRES SAME SCHEMA

Friday, May 22, 2009

CUSTOM HANDLERS

THAR BE DRAGONS

Friday, May 22, 2009

CUSTOM HANDLERS

ALLOWS FOR DIFFERENT SCHEMAS FOR THE SAME DATA EXAMPLE: LEGACY SCHEMA VS NEW SCHEMA DIFFERENT FOCUSED TARGET RELATIONS

THAR BE DRAGONS

WARNING: NEED TO UNDERSTAND INTERNAL GOLCONDE FLOW

Friday, May 22, 2009

CLIENT-BASED FLOW

Friday, May 22, 2009

CLIENT-BASED FLOW

CLIENT APPLICATION ENQUEUES

Friday, May 22, 2009

CLIENT-BASED FLOW

CLIENT APPLICATION ENQUEUES GOLCONDE PERFORMS TRANSACTION ON CANONICAL DESTINATION RELATION

Friday, May 22, 2009

CLIENT-BASED FLOW

CLIENT APPLICATION ENQUEUES GOLCONDE PERFORMS TRANSACTION ON CANONICAL DESTINATION RELATION SERIALLY DISTRIBUTES TO TARGET QUEUES

Friday, May 22, 2009

CLIENT-BASED FLOW

Friday, May 22, 2009

CLIENT-BASED FLOW

TARGET HANDLER DEQUEUES

Friday, May 22, 2009

CLIENT-BASED FLOW

TARGET HANDLER DEQUEUES PERFORMS TRANSACTION ON TARGET RELATIONS

Friday, May 22, 2009

TRIGGER-BASED USAGE

Friday, May 22, 2009

TRIGGER-BASED USAGE

REQUIRES PL/PYTHON

Friday, May 22, 2009

TRIGGER-BASED USAGE

REQUIRES PL/PYTHON TRIGGER-UTIL.PY BUILDS AND INSTALLS TRIGGERS

Friday, May 22, 2009

TRIGGER-BASED USAGE

REQUIRES PL/PYTHON TRIGGER-UTIL.PY BUILDS AND INSTALLS TRIGGERS TRIGGER IS INSTALLED ON THE CANONICAL TABLE

Friday, May 22, 2009

TRIGGER-BASED USAGE

REQUIRES PL/PYTHON TRIGGER-UTIL.PY BUILDS AND INSTALLS TRIGGERS TRIGGER IS INSTALLED ON THE CANONICAL TABLE NO MODIFICATION OF CLIENT APPLICATION REQUIRED

Friday, May 22, 2009

TRIGGER-BASED FLOW

Friday, May 22, 2009

TRIGGER-BASED FLOW

FIRES AFTER INSERT/UPDATE/ DELETE

Friday, May 22, 2009

TRIGGER-BASED FLOW

FIRES AFTER INSERT/UPDATE/ DELETE ENQUEUES INTO TARGET QUEUES

Friday, May 22, 2009

TRIGGER-BASED FLOW

FIRES AFTER INSERT/UPDATE/ DELETE ENQUEUES INTO TARGET QUEUES GOLCONDE PERFORMS TRANSACTIONS ON THE TARGET RELATIONS

Friday, May 22, 2009

CONFIGURATION

HTTP://WWW.YAML.ORG/

Friday, May 22, 2009

CONFIGURATION

YAML BASED

HTTP://WWW.YAML.ORG/

Friday, May 22, 2009

CONFIGURATION

YAML BASED PARAMETERS FOR:

HTTP://WWW.YAML.ORG/

Friday, May 22, 2009

CONFIGURATION

YAML BASED PARAMETERS FOR: LOGGING

HTTP://WWW.YAML.ORG/

Friday, May 22, 2009

CONFIGURATION

YAML BASED PARAMETERS FOR: LOGGING HTTP DAEMON FOR REALTIME MONITORING & VISUALIZATION

HTTP://WWW.YAML.ORG/

Friday, May 22, 2009

CONFIGURATION

YAML BASED PARAMETERS FOR: LOGGING HTTP DAEMON FOR REALTIME MONITORING & VISUALIZATION DESTINATION / TARGET GROUPS
HTTP://WWW.YAML.ORG/

Friday, May 22, 2009

STATISTICS & VISUALIZATION

Friday, May 22, 2009

STATISTICS & VISUALIZATION

BUILT-IN HTTP SERVER

Friday, May 22, 2009

STATISTICS & VISUALIZATION

BUILT-IN HTTP SERVER STATS REQUESTS RETURNS JSON USE WITH STAPLR, NAGIOS, ETC

Friday, May 22, 2009

STATISTICS & VISUALIZATION

BUILT-IN HTTP SERVER STATS REQUESTS RETURNS JSON USE WITH STAPLR, NAGIOS, ETC INTERNAL REAL-TIME VISUALIZATION

Friday, May 22, 2009

DEMONSTRATION

Friday, May 22, 2009

CURRENT STATUS

Friday, May 22, 2009

CURRENT STATUS

0.5 BETA RELEASE - 03/02

Friday, May 22, 2009

CURRENT STATUS

0.5 BETA RELEASE - 03/02 0.6 BETA COMING SOON ADDS REAL-TIME STATS SERVER BUG FIXES

Friday, May 22, 2009

ROADMAP

Friday, May 22, 2009

ROADMAP

TWO-PHASE COMMIT LIKE BEHAVIOR ALL TRANSACTIONS MUST COMMIT OR ROLLBACK ROLLBACK IS DEFINED BY ROLLBACK DATA PACKETS IN AUTOSQL OR HANDLERS

Friday, May 22, 2009

ROADMAP

TWO-PHASE COMMIT LIKE BEHAVIOR ALL TRANSACTIONS MUST COMMIT OR ROLLBACK ROLLBACK IS DEFINED BY ROLLBACK DATA PACKETS IN AUTOSQL OR HANDLERS ADDITIONAL QUEUE SERVERS AND PROTOCOLS

Friday, May 22, 2009

ROADMAP

TWO-PHASE COMMIT LIKE BEHAVIOR ALL TRANSACTIONS MUST COMMIT OR ROLLBACK ROLLBACK IS DEFINED BY ROLLBACK DATA PACKETS IN AUTOSQL OR HANDLERS ADDITIONAL QUEUE SERVERS AND PROTOCOLS “NATIVE” CLIENT CLASSES IN PHP, PYTHON

Friday, May 22, 2009

QUESTIONS?

CONTACT INFO: GMR@MYYEARBOOK.COM TWITTER: @CRAD HTTP://GAVINROY.COM

Friday, May 22, 2009

You're Reading a Free Preview

Download
scribd
/*********** DO NOT ALTER ANYTHING BELOW THIS LINE ! ************/ var s_code=s.t();if(s_code)document.write(s_code)//-->