Postgresql v200b

..
Postgresql
,
2011
Creative Commons Attribution-Noncommercial 2.5
(, )
:
PostgreSQL: . (Sad Spirit)

borz_o@cs.msu.su,
http://www.phpclub.ru/detail/store/pdf/postgresql-performance.pdf
PostgreSQL Slony-I,
Eugene Kuzin eugene@kuzin.net,
http://www.kuzin.net/work/sloniki-privet.html
Londiste , Sergey Konoplev gray.ru@gmail.com,

http://gray-hemp.blogspot.com/2010/04/londiste.html
pgpool-II, Dmitry Stasyuk,

http://undenied.ru/2009/03/04/uchebnoe-rukovodstvo-po-pgpool-ii/
PostgreSQL PL/Proxy,
dmitry.chirkin@gmail.com,
http://habrahabr.ru/blogs/postgresql/45475/
Hadoop, wordpress@insight-it.ru,
http://www.insight-it.ru/masshtabiruemost/hadoop/
Up and Running with HadoopDB, Padraig O'Sullivan,

http://posulliv.github.com/2010/05/10/hadoopdb-mysql.html
PostgreSQL: Skype, ,
http://postgresmen.ru/articles/view/25
Streaming Replication,
http://wiki.postgresql.org/wiki/Streaming_Replication
, , - ?, Den Golotyuk,
http://highload.com.ua/index.php/2009/05/06/-/
2
2.1
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . . .

2.2
2.3
2.4
. . . . . . . .
10
. . . . . . . . . . . . . . . . . . . . . . . . .
10
. . . . . . . . . . . . . . . . . . . . . . .
11
. . . . . . . . . . .
15
. . . . . . . . . . . . . . . . . . . . . .
17
. . . . . . . . . . . . . . . . . . . . . . . . . .
18
. . . . . . . . . . . . . . . . . . .
19
. . . . . . .
19
. . . . . . . . . . . . . . . . . . . . . . . . .
20
. . . . . . . . . . . . . . . . . . . . . . .
20

(1), 2 . . . . . . . . . . . . . . . . . . . . .
20
Web , 2
. . . . . . . . . . . . . . . . . . . . . . . . . . .
21
Web , 8
. . . . . . . . . . . . . . . . . . . . . . . . . . .
21
2.5
: pgtune
. .
21
2.6
. . . . . . . . . . . . . . . . .
22
2.7
. . . . . . . . . . . . . . . . . . .
22
23
. . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . .
26
. . . . . . . . . . . . . . .
26
pgFouine
. . . . . . . . . .
28
. . . . . . . . . . . . . . . . . . . . . . . . . . . . .
29
30
3.1
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.2
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
31
3.3
. . . . . . . . . . . . . . . . . . . . .
32
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
32
. . . . . . . . . . . . . . . . . . . . . . . . . . . .
34
35
. . . . . . . . . . . . . . . . . . . . .
constraint_exclusion
3.4
30
. .
36
. . . . . . . . . . . . . . . . . . . . . . . . . . . . .
38
39
4.1
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
39
4.2
Slony-I . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
41
4.3
4.4
4.5
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
41
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
42
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
42
. . . . . . . . . . . . . . . . . . . . . . . . . . .
47
. . . . . . . . . . . . . . . . . . . .
49
Londiste . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
52
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
52
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
52
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
54
58
. . . . . . . . . . . . . . . . . . . .
59
Streaming Replication ( ) . . . . . . . . .
60
60
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
60
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
60

Bucardo
. . . . . . . . . . . . . . . . . . . . . . . . . . .
66
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
67
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
67
4.6
4.7
. . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
67
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
67
70
. . . . . . . . . . . . . . . . . . . . . . . . . . .
RubyRep . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
70
70
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
71
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
72
. . . . . . . . . . . . . . . . . . . .
73
. . . . . . . . . . . . . . . . . . . . . . . . . . . . .
74
76
5.1
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
76
5.2
PL/Proxy
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
78
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
78
5.3
5.4
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
79
? . . . . . . . . . . . . . . . . . . . . . . . . .
82
HadoopDB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
83
87
. . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . .
98
. . . . . . . . . . . . . . . . . . . . . . . . . . . . .
99
6 PgPool-II
100
6.1
6.2
! . . . . . . . . . . . . . . . . . . . . . . . . . . 101
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
pgpool-II
. . . . . . . . . . . . . . . . . . . . . . . . 101
. . . . . . . . . . . . . . . . . . . . . . . 102
PCP . . . . . . . . . . . . . . . . . . . . . . 103

/ pgpool-II
6.3
. . . . . . . . . . . . . . . . . . 104
. . . . . . . . . . . . . . . . . . . 104
. . . . . . . . . . . . . . . . . . . . . 105

. . . . . . . . . . . . . . . . . . . . . . 105
. . . . . . . . . . . . . . . . . . . . . . . 106
6.4
. . . . . . . . . . . . . . . . 107
. . . . . . . . . . . . . . . . 107
SystemDB
. . . . . . . . . . . . . . . . . . . . . . . 108
. . . . . . . . . . . . 111

. . . . . . . . . . . . . . . . . . 112

6.5
Master-slave
. . . . . . . . . . . . . . . . 113
. . . . . . . . . . . . . . . . . . . . . . . . 114
Streaming Replication ( ) . . . . . . . . . 114

6.6
. . . . . . . . . . . . . . . . . . . . . . . 115
Streaming Replication ( ) . . . . . . . . . 116
6.7
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
119
7.1
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
7.2
PgBouncer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
7.3
PgPool-II vs PgBouncer
. . . . . . . . . . . . . . . . . . . . . . 120
8 PostgreSQL
122
8.1
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
8.2
Pgmemcache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
9
9.1
130
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
9.2
PostGIS
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
9.3
PostPic
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
9.4
OpenFTS
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
9.5
PL/Proxy
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
9.6
Texcaller . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
9.7
Pgmemcache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
9.8
Prex
9.9
pgSphere . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
9.10 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
10 PostgreSQL
10.1
133
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
10.2 SQL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133

SQL
10.3
. . . . . . . . . . . . . . . . . 135
. . . . . . . . . . . . . . . . 136
10.4
. . . . . . . . . . . . . . 137
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138
10.5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138
11 PostgreSQL
11.1
139
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
. . . . . . . . . . . . . . . . . . . . . . . . . . . 139
11.2 . . . . . . . . . . . . . . . . . . . . . 140
. . . . . . . . . . . . . . . . . . . . . . . . . . 140
11.3 . . . . . . . . . . . . . . . . . . . . . 141
. . . . . . . . . . . . . . . . . . . . . . . . . . 141
12 (Performance Snippets)
12.1
12.2
142
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
. . . . . . . . . . . . . . . . . 143
. . . . . . . . . . . . . . . . . . 143
count . . . . . . . . . . . . . . . . . . . . . . . . . . 144
-
. . . . . . . 145
. . . . . . . . . . . . . . . . . . 146
. . . . . . . . . . . . . . . . . . . . . . . . . . . 146

,
. . . . . . 149
. . . . . . . . . . . 149
LIKE . . . . . . . . . . . . . . . . . . . . . . . . . . . 150
,
, .
PostgreSQL.
PostgreSQL, . ,
.
2

,
.

2.1
, , . ,
.

;
.
,
.
, ,
?
, ?
,
.
PostgreSQL. , PostgreSQL
, , FAQ. postgresql-performance, . , ,
. : - .
2.1.

PostgreSQL ,
. . : ,
PostgreSQL, . (
) , . PostgreSQL, ,
. , , ..
PostgreSQL :
;
, ;
;
, (, ).

PostgreSQL, , .
.
7.1 ,
.
7.2 :

VACUUM, ;
ANALYZE, ,
;
.
7.4 ( IN/NOT IN).
8.0 , , CHECKPOINT VACUUM .
8.1
, MIN() MAX(),
pg_autovacuum (), .
8.2 SQL , .
2.2.
8.3 , SQL/XML
, .
8.4 , , , EXISTS/NOT EXISTS .
9.0 , VACUUM/VACUUM FULL

, .
, 8.4.

, ,
,
. , ,
. ,
, , ? , ,
.
,
, .
, , :
.
(
).
(, ,
).
(.
3.4).
2.2
, . postgresql.conf
.
10
2.2.
: shared_buers
PostgreSQL
. ,
,
, .
,
. , , . , ,
.

,
, .
: , PostgreSQL, PostgreSQL ,
. , PostgreSQL
, , . ,
, , .
, shared_buers,
, ,
, .
8
2 . ,
, , , ,
. , , .
, .
:
4 (512)
256512 : 1632
(20484096)
14 : 64256
(819232768)
11
2.2.
ipcs (, free
vmstat). 1,2 2
, . ,
, . , .
PostgreSQL , :
http://developer.postgresql.org/docs/postgres/kernel-resources.html
, :
Laptop, Celeron processor, 384 RAM, 25 : 12

Athlon server, 1 RAM,
10 : 200
Quad PIII server, 4 RAM, 40 , 150 , : 1
Quad Xeon server, 8 RAM, 200 , 300 ,

: 2
: work_mem
sort_mem, ,
, , . , work_mem (
). :
( , ,
, shared_buers) ,
.
, .
, .
work_mem postgresql.conf.
1 . 1024.
24% . -
work_mem, , , 512 2048 . ,
work_mem 500
. , , ,
, -
12
2.2.
. , 14 32128 MB.
VACUUM:
maintenance_work_mem
PostgreSQL 7.x vacuum_mem.
, VACUUM, ANALYZE, CREATE
INDEX, .
, ,
.
50 75% , , 32 256 .
, work_mem.
. , 14
128512 MB.
Free Space Map: VACUUM FULL

( PostgreSQL) :
, , , , ( );
( UPDATE DELETE)
( ).
, PostgreSQL
VACUUM ( 3.1.1).
7.2 VACUUM . 7.2, VACUUM , SELECT, INSERT,
UPDATE DELETE . VACUUM FULL.
, , , ,
.
:
max_fsm_relations
,
.
VACUUM. max_fsm_relations

( ).
/
13
2.2.
max_fsm_pages
,
,
.
, ,
VACUUM. ,
VACUUM VERBOSE ANALYZE ,
. max_fsm_pages , .
FSM, VACUUM
, VACUUM FULL,
.
8.4 fsm ,
Free Space Map , .

temp_buers
, .
16 .
max_prepared_transactions
(PREPARE
TRANSACTION). 5.
vacuum_cost_delay
,
, , I/O VACUUM, .
,
vacuum_cost_delay 0. 50
200 . vacuum_cost_page_hit
vacuum_cost_page_limit. VACUUM,
. (Jan Wieck) , delay 200,
page_hit 6 100 VACUUM
80%, .
max_stack_depth
, , . , , . 24 MB.
14
2.2.
max_les_per_process
,
. , Too many open
les.

PostgreSQL : ( ) , ,
.
: . ,
:
( checkpoint_segments, 3)
, ( checkpoint_timeout, ,
300).
,
, .
:
checkpoint_segments
,
.
- .
(checkpoint_segments).
( 16 )
. ,
, .
12 256 , (warning) , , .
, , (checkpoint_segments
.
checkpoint_warning ( ):
, .
2
15
2.2.
* 2 + 1) * 16 , , . , 32, 1
.
, ,
.
fsync

o fsync.
,
. : ,
, !
,
.
.

commit_delay ( , 0 ) commit_sib-
lings (5 )

.
commit_siblings ,
commit_delay. ,
, . , .
wal_sync_method
,
. fsync=o, .
:

open_datasync open()
O_DSYNC
fdatasync fdatasync() commit
fsync_writethrough fsync() commit

fsync fsync() commit
open_sync open() O_SYNC
. , .
16
2.2.
full_page_writes
o, fsync=o. ,
on, PostgreSQL
. , ,
. ,
. ,
. full_page_writes , (
checkpoint_interval).
wal_buers
SHARED MEMORY
. 256512 ,
. ,
14 2561024 .

. 3 ,
:
default_statistics_target
, ANALYZE
(. 3.1.2). , , .
ALTER TABLE . . . SET STATISTICS.
eective_cache_size
PostgreSQL
,
.
1,5 , shared_buers
32 , eective_cache_size 800 .

,
3
4
17
2.2.
700 , PostgreSQL ,
merge joins. eective_cache_size
200 , , .
eective_cache_size
2/3 ; RAM
, .
random_page_cost
, . 3.0, 2.5
2.0.
, .
. , , (sequential scans) (index
scans), . , , ,
. . random_page_cost 2.0;
, random_page_cost ,
.

PostgreSQL , .

, ,
. , true/false:
track_counts
. -
, autovacuum . , (
autovacuum).
track_functions
track_activities
18
2.3.
. ,
, ,
.
, , .
, : , (
autovacuum ).
2.3
, . , , , .
PostgreSQL , ,
. ,
, .
, ,
, noatime .

, .
, .
, , ,

, .
:
(!).

, , :
5
6
19
2.4.
pg_clog pg_xlog,
, .
.
.
, , ,
, , , .
2.4

.
PostgreSQL .
RAM ;
shared_buers = 1/8 RAM ( 1/4);

work_mem 1/20 RAM;
maintenance_work_mem 1/4 RAM;
max_fsm_relations * 1.5;
max_fsm_pages max_fsm_relations * 2000;
fsync = true;
wal_sync_method = fdatasync;
commit_delay = 10 100 ;
commit_siblings = 5 10;
eective_cache_size = 0.9 cached,
free;
random_page_cost = 2 cpu, 4 ;
cpu_tuple_cost = 0.001 cpu, 0.01 ;
cpu_index_tuple_cost = 0.0005 cpu, 0.005 ;
autovacuum = on;
autovacuum_vacuum_threshold = 1800;
autovacuum_analyze_threshold = 900;

(1), 2
maintenance_work_mem = 128MB
eective_cache_size = 512MB
work_mem = 640kB
20
2.5.
: pgtune
wal_buers = 1536kB
shared_buers = 128MB
max_connections = 500
Web
, 2
maintenance_work_mem = 128MB;
checkpoint_completion_target = 0.7
eective_cache_size = 1536MB
work_mem = 4MB
wal_buers = 4MB
checkpoint_segments = 8
shared_buers = 512MB
Web
, 8
maintenance_work_mem = 512MB
checkpoint_completion_target = 0.7
eective_cache_size = 6GB
work_mem = 16MB
wal_buers = 4MB
checkpoint_segments = 8
shared_buers = 2GB
2.5
: pgtune
PostgreSQL Gregory Smith -
pgtune
. Linux .
, . :
Listing 2.1: Pgtune
1
2
pgtune -i $PGDATA / postgresql . conf \

-o $PGDATA / postgresql . conf . pgtune
7
http://pgtune.projects.postgresql.org/
21
2.6.
-i, --input-config postgresql.conf,

--output-config postgresql.conf.
-o,
-M, --memory
. , pgtune
.
-T, --type
. : DW, OLTP, Web,
Mixed, Desktop.
-c, --connections
, .
, pgtune PostgreSQL.
, , , .
2.6
:
1. , . :
a) . .
b) , .
2. .
3. .
4. .

,
. ( cron)
.
ANALYZE
.
.
22
2.6.
VACUUM ANALYZE.
, ,
,
ANALYZE.
.
REINDEX
REINDEX . :
;
.
. , ,
. PostgreSQL
, . ,
.
- , REINDEX. :
REINDEX, VACUUM FULL, ,
, .

, . , ,
, . .
, , :
, ,
. , ,
.
. ,
.
,
, , ,
, .
EXPLAIN [ANALYZE]
EXPLAIN [] , PostgreSQL . EXPLAIN ANALYZE []
23
2.6.
.
, .
:
(seq scan).

(nested loop).
EXPLAIN ANALYZE: ?
,
.
,
. , ,
, - ,
,
. 80%
,
.
EXPLAIN ANALYZE
, . ,
SET enable_seqscan=false;
,
, , .
postgresql.conf ! ,
!

. :
pg_stat_user_tables
, ,
, ,
.
EXPLAIN ANALYZE DELETE . . .

24
2.6.
pg_stat_user_indexes ,
, , ( , ,
).
pg_statio_user_tables
, , , (. 2.1.1), , , TOAST.
,
).
.
, , , ,
PRIMARY KEY UNIQUE.

.
, , .
PostgreSQL

/ , ,
. , , foo foo_name,
foo_name = '',
.
CREATE INDEX foo_name_first_idx

ON foo ((lower(substr(foo_name, 1, 1))));

SELECT * FROM foo

WHERE lower(substr(foo_name, 1, 1)) = '';
.
(partial indexes)
WHERE. , ,
scheta uplocheno boolean. ,
uplocheno = false , uplocheno = true,
.
25
2.6.
CREATE INDEX scheta_neuplocheno ON scheta (id)

WHERE NOT uplocheno;

SELECT * FROM scheta WHERE NOT uplocheno AND ...;

, ,
WHERE, .

PostrgeSQL , PostgreSQL ,
.
, ,
, ,
;
.
, : ,
.

, , . ,

, .
SELECT count(*) FROM < >

count() : , ,
. (
!) , ,
9 RULE PostgreSQL SQL, ,

,
26
2.6.
, .
Listing 2.2: SQL
SELECT count (*) FROM foo ;

foo,
.
, , .
- :
10
1. ,
, ANALYZE:
Listing 2.3: SQL
SELECT reltuples FROM pg_class WHERE relname = ' foo ';
2. , , ,
. ,
. ,
.
3. , (cron).
DISTINCT
DISTINCT .
GROUP BY DISTINCT. GROUP BY
, ,
DISTINCT.
Listing 2.4: DISTINCT
1
2
3
4
5
7
postgres =# select count (*) from ( select distinct i from g ) a ;

count
-- ----19125
(1 row )
Time : 580 ,553 ms
10000 ,
50000 !
10
27
2.6.
10
11
12
13
14
16
postgres =# select count (*) from ( select distinct i from g ) a ;

count
-- ----19125
(1 row )
Time : 36 ,281 ms
Listing 2.5: GROUP BY
1
2
3
4
5
7
10
11
12
13
14
16
postgres =# select count (*) from ( select i from g group by i ) a ;

count
-- ----19125
(1 row )
Time : 26 ,562 ms
postgres =# select count (*) from ( select i from g group by i ) a ;
count
-- ----19125
(1 row )
Time : 25 ,270 ms
pgFouine
pgFouine
11
log- PostgreSQL,
log- PostgreSQL. pgFouine , . pgFouine PHP - ,

GNU General Public License.
,
log- .
pgFouine PostgreSQL
log-:
syslog
Listing 2.6: pgFouine
11
http://pgfouine.projects.postgresql.org/
28
2.7.
1
2
3
log_destination = ' syslog '

redirect_stderr = off
silent_mode = on
, n :
1
2
3
log_min_duration_statement = n
log_duration = off
log_statement = ' none '
log_min_duration_statement
0. , -1.
pgFouine .
HTML- :
pgfouine . php - file your / log / file . log > your - report . html
10 :
pgfouine . php - file your / log / file . log - top 10 - format text
, ,
http://pgfouine.projects.postgresql.org.
2.7
, PostgreSQL .
, . .
29
- ,
. , ,
.

3.1
(partitioning, )
(, ) .
, .
( ).
(.. ,
). ( 5. . . 10 ) 40. . . 50% ,
( ), . . .
. , ( )
, ( ). , ,
. SELECT *
FROM articles ORDER BY id DESC LIMIT 10
, .
, :
(, ,
) -
.

(DROP TABLE , DELETE).
30
3.2.
3.2
PostgreSQL :
(range) , ,
. , .
(list)
.
,
:
, . .
,
.
, .
,
. ,
. :
Listing 3.1:
1
2
CHECK ( outletID BETWEEN 100 AND 200 )

CHECK ( outletID BETWEEN 200 AND 300 )
,
200.
( ), .
, .
, constraint_exclusion postgresql.conf.
, .
31
3.3.
3.3
. , ,
.
. , , ()
(). ,
3 . ,
. , , , ,
( ). .
.
, :
Listing 3.2:
1
2
3
4
5
6
7
CREATE TABLE my_logs (

id
SERIAL PRIMARY KEY ,
user_id
INT NOT NULL ,
logdate
TIMESTAMP NOT NULL ,
data
TEXT ,
some_state
INT
);
, .
.
my_logs,
. ():
Listing 3.3:
1
2
3
4
5
6
7
8
9
10
11
CREATE TABLE my_logs2010m10

CHECK ( logdate >= DATE
' 2010 -11 -01 ' )
) INHERITS ( my_logs ) ;
' 2010 -12 -01 ' )
' 2011 -01 -01 ' )
' 2010 -02 -01 ' )
(
' 2010 -10 -01 ' AND logdate < DATE
(
(
(
32
3.3.
12
my_logs2010m10, my_logs2010m11
.., ( ).
CHECK , ( , !).
logdate,
:
Listing 3.4:
1
2
3
4
CREATE INDEX my_logs2010m10_logdate

( logdate ) ;
( logdate ) ;
( logdate ) ;
( logdate ) ;
ON my_logs2010m10
ON my_logs2010m11
ON my_logs2010m12
ON my_logs2011m01
,
.
Listing 3.5:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
CREATE OR REPLACE FUNCTION my_logs_insert_trigger ()

RETURNS TRIGGER AS $$
BEGIN
IF ( NEW . logdate >= DATE ' 2010 -10 -01 ' AND
NEW . logdate < DATE ' 2010 -11 -01 ' ) THEN
INSERT INTO my_logs2010m10 VALUES ( NEW .*) ;
ELSIF ( NEW . logdate >= DATE ' 2010 -11 -01 ' AND
ELSE
RAISE EXCEPTION ' Date out of range . Fix the
my_logs_insert_trigger () function ! ';
END IF ;
RETURN NULL ;
END ;
$$
LANGUAGE plpgsql ;
: logdate,
.
33
3.3.
.
:
Listing 3.6:
1
2
3
CREATE TRIGGER insert_my_logs_trigger

BEFORE INSERT ON my_logs
FOR EACH ROW EXECUTE PROCEDURE my_logs_insert_trigger () ;
.
my_logs:
Listing 3.7:
1
2
3
INSERT INTO my_logs ( user_id , logdate , data , some_state )

VALUES (1 , ' 2010 -10 -30 ' , ' 30.10.2010 data ' , 1) ;
VALUES (2 , ' 2010 -11 -10 ' , ' 10.11.2010 data2 ' , 1) ;
VALUES (1 , ' 2010 -12 -15 ' , ' 15.12.2010 data3 ' , 1) ;
:
Listing 3.8:
1
2
3
4
partitioning_test =# SELECT * FROM ONLY my_logs ;

id | user_id | logdate | data | some_state
-- - -+ - - - - - - - - -+ - - - - - - - - -+ - - - - - -+ - - - - - - - - - - - (0 rows )
.
:
Listing 3.9:
1
2
3
4
5
6
7
partitioning_test =# SELECT * FROM my_logs ;

id | user_id |
logdate
|
data
|
some_state
-- - -+ - - - - - - - - -+ - - - - - - - - - - - - - - - - - - - - -+ - - - - - - - - - - - - - - - - - -+ - - - - - - - - - - - 1 |
1 | 2010 -10 -30 00:00:00 | 30.10.2010 data |
1
2 |
2 | 2010 -11 -10 00:00:00 | 10.11.2010 data2 |
1
3 |
1 | 2010 -12 -15 00:00:00 | 15.12.2010 data3 |
1
(3 rows )
. , :
34
3.3.
Listing 3.10:
1
2
3
4
5
7
8
9
10
11
partitioning_test =# Select * from my_logs2010m10 ;

id | user_id |
logdate
|
data
|
some_state
-- - -+ - - - - - - - - -+ - - - - - - - - - - - - - - - - - - - - -+ - - - - - - - - - - - - - - - - -+ - - - - - - - - - - - 1 |
1 | 2010 -10 -30 00:00:00 | 30.10.2010 data |
1
(1 row )
partitioning_test =# Select * from my_logs2010m11 ;
id | user_id |
logdate
|
data
|
some_state
-- - -+ - - - - - - - - -+ - - - - - - - - - - - - - - - - - - - - -+ - - - - - - - - - - - - - - - - - -+ - - - - - - - - - - - 2 |
2 | 2010 -11 -10 00:00:00 | 10.11.2010 data2 |
1
(1 row )
! .
my_logs :
Listing 3.11:
1
2
3
4
5
7
8
9
10
11
12
partitioning_test =# SELECT * FROM my_logs WHERE user_id = 2;

id | user_id |
logdate
|
data
|
some_state
-- - -+ - - - - - - - - -+ - - - - - - - - - - - - - - - - - - - - -+ - - - - - - - - - - - - - - - - - -+ - - - - - - - - - - - 2 |
2 | 2010 -11 -10 00:00:00 | 10.11.2010 data2 |
1
(1 row )
partitioning_test =# SELECT * FROM my_logs WHERE data LIKE
' %0.1% ';
id | user_id |
logdate
|
data
|
some_state
-- - -+ - - - - - - - - -+ - - - - - - - - - - - - - - - - - - - - -+ - - - - - - - - - - - - - - - - - -+ - - - - - - - - - - - 1 |
1 | 2010 -10 -30 00:00:00 | 30.10.2010 data |
1
2 |
2 | 2010 -11 -10 00:00:00 | 10.11.2010 data2 |
1
(2 rows )

. . ,
2008 , 10 . :
Listing 3.12:
DROP TABLE my_logs2008m10 ;

35
3.3.
DROP TABLE , DELETE. ,

, ,
,
:
Listing 3.13:
ALTER TABLE my_logs2008m10 NO INHERIT my_logs ;

, .
constraint_exclusion
constraint_exclusion ,
. , :
Listing 3.14: constraint_exclusion OFF
1
2
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
partitioning_test =# SET constraint_exclusion = off ;

partitioning_test =# EXPLAIN SELECT * FROM my_logs WHERE
logdate > ' 2010 -12 -01 ';
QUERY PLAN
-- - -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --- -- -- -- -Result ( cost =6.81..104.66 rows =1650 width =52)
-> Append ( cost =6.81..104.66 rows =1650 width =52)
-> Bitmap Heap Scan on my_logs ( cost =6.81..20.93
rows =330 width =52)
Recheck Cond : ( logdate > ' 2010 -12 -01
00:00:00 ' :: timestamp without time zone )
-> Bitmap Index Scan on my_logs_logdate
( cost =0.00..6.73 rows =330 width =0)
Index Cond : ( logdate > ' 2010 -12 -01
-> Bitmap Heap Scan on my_logs2010m10 my_logs
( cost =6.81..20.93 rows =330 width =52)
-> Bitmap Index Scan on my_logs2010m10_logdate
( cost =0.00..6.73 rows =330 width =0)
( cost =6.81..20.93 rows =330 width =52)
( cost =0.00..6.73 rows =330 width =0)
36
3.3.
19
20
21
22
23
24
25
26
27
28

( cost =6.81..20.93 rows =330 width =52)
( cost =0.00..6.73 rows =330 width =0)
( cost =6.81..20.93 rows =330 width =52)
( cost =0.00..6.73 rows =330 width =0)
(22 rows )
EXPLAIN,
, ,
logdate > 2010-12-01 , , .
constraint_exclusion:
Listing 3.15: constraint_exclusion ON
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
partitioning_test =# SET constraint_exclusion = on ;

SET
partitioning_test =# EXPLAIN SELECT * FROM my_logs WHERE
logdate > ' 2010 -12 -01 ';
QUERY PLAN
-- - -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --- -- -- -- -Result ( cost =6.81..41.87 rows =660 width =52)
-> Append ( cost =6.81..41.87 rows =660 width =52)
-> Bitmap Heap Scan on my_logs ( cost =6.81..20.93
rows =330 width =52)
-> Bitmap Index Scan on my_logs_logdate
( cost =0.00..6.73 rows =330 width =0)
( cost =6.81..20.93 rows =330 width =52)
( cost =0.00..6.73 rows =330 width =0)
(10 rows )
37
3.4.
, ,
, . constraint_exclusion
, , CHECK , , . 8.4 PostgreSQL
constraint_exclusion on, o partition. ( ) constraint_exclusion on, o,
partition, CHECK .
3.4

.
, . ,
, ( ) 50%
.
38
, .
,
.

4.1
(. replication) (, ). ,
. , ,
. .
, ,
. , . (,
).
,
, - ( , , ).
, .
, , , ( ,
). , . ,
39
4.1.
,

.
,
( ). ,
, , , ,
.
(, , ). ,
, ( )
. ,
, . , ,
X, B , , Y X.
Y, Y (, - ), B
Y , , ,
.
, Y , X .
. , ,
,
( ) .

: ,
. , , .
,
. , . ,
, ,
.
,

.
PostgreSQL , , . (, ).
40
4.2.
Slony-I
Slony-I1 Master-Slave , (cascading) (failover). Slony-I PostgreSQL INSERT/ DELETE/UPDATE

.
PGCluster2
Multi-Master .
, .
pgpool-I/II3 PostgreSQL ( II ). :

( ,
stand-by );
online-;
pooling ;
;
SELECT- postgresql-;
.
Bucardo4
, Multi-
Master Master-Slave , .
Londiste5
6
Master-Slave .
Skytools . , Slony-I.
Mammoth Replicator7 Multi-Master .

Postgres-R8 Multi-Master .
RubyRep9 Ruby, Multi-Master , PostgreSQL MySQL.
, , ,
PostgreSQL.
4.2 Slony-I
Slony , PostgreSQL . Slony
http://www.slony.info/
http://pgfoundry.org/projects/pgcluster/
3 http://pgpool.projects.postgresql.org/
4 http://bucardo.org/
5 http://skytools.projects.postgresql.org/doc/londiste.ref.html
6 http://pgfoundry.org/projects/skytools/
7 http://www.commandprompt.com/products/mammothreplicator/
8 http://www.postgres-r.org/
9 http://www.rubyrep.org/
1
2
41
4.2.
Slony-I
Postgre INSERT/ DELETE/UPDATE

.
Slony , slony slonik. slonik-,
slon .
, slon , .
slonik-e
slonik stdin.
slonik-a , , ,
slonik syntax error, .
. .
Ubuntu :
Listing 4.1:
sudo aptitude install slony1 - bin
customers
( , ).

: customers
master_host: customers_master.com
slave_host_1: customers_slave.com
cluster name ( ): customers_rep
master-
Postgres,
Slony. , ,
slony.
Listing 4.2: master-
1
2
3
pgsql@customers_master$ createuser -a -d slony

pgsql@customers_master$ psql -d template1 -c " alter \
user slony with password ' slony_user_password '; "
42
4.2.
Slony-I

slony, slon. , ( slon)
.
slave-
,
Internet ( ), PostgreSQL , . , :
Listing 4.3: slave-
1
2
anyuser@customers_slave$ psql -d customers \

-h customers_master . com -U slony
- ( , ). - , rewall-a, pg_hba.conf, $PGDATA.
slave- PostgreSQL.
, Postgres up and ready,
- , (
postmaster):
Listing 4.4: slave-
1
2
3
4
5
6
pgsql@customers_slave$ rm - rf $PGDATA
pgsql@customers_slave$ mkdir $PGDATA
pgsql@customers_slave$ initdb -E UTF8 -D $PGDATA
pgsql@customers_slave$ createuser -a -d slony
pgsql@customers_slave$ psql -d template1 -c " alter \
user slony with password ' slony_user_password '; "
postmaster.
! . !
Listing 4.5: slave-
1
2
3
pgsql@customers_slave$ createuser -a -d customers_owner

pgsql@customers_slave$ psql -d template1 -c " alter \
user customers_owner with password ' customers_owner_password '; "
customers_master,
-h customers_slave, slave.
slave, master, Slony.
43
4.2.
Slony-I
plpgsql slave
slony.
(slony_user_password).
:
Listing 4.6: plpgsql slave
1
2
3
4
slony@customers_master$ createdb -O customers_owner \

-h customers_slave . com customers
slony@customers_master$ createlang -d customers \
-h customers_slave . com plpgsql
! , replication set primary key. -
, primary key
ALTER TABLE ADD PRIMARY KEY.
primary key ,
serial (ALTER TABLE ADD COLUMN),
. table add
key slonik-a.
. slave:
1
2
slony@customers_master$ pg_dump -s customers | \

psql -U slony -h customers_slave . com customers
pg_dump -s .
pg_dump -s customers , psql -U
slony -h customers_slave.com customers (slony_user_pass).
: -
Slony ( make install), sl_*,
. , :
( 5)
:-) slave :
1
2
3
4
5
6
7
slonik << EOF

cluster name = customers_slave ;
node Y admin conninfo = ' dbname = customers
host = customers_master . com
port =5432 user = slony password = slony_user_pass ';
uninstall node ( id = Y ) ;
echo ' okay ';
EOF
44
4.2.
Slony-I
Y . . : , cluster
name - , T1 (default).
uninstall.
( ), uninstall ( master slave).

PgSQL
, - ,
.
- :
Listing 4.9:
1
3
5
6
8
9
11
12
14
16
17
18
19
20
21
22
23
25
27
28
30
32
33
34
35
# !/ bin / sh
CLUSTER = customers_rep
DBNAME1 = customers
DBNAME2 = customers
HOST1 = customers_master . com
HOST2 = customers_slave . com
PORT1 =5432
PORT2 =5432
SLONY_USER = slony
slonik << EOF
cluster name = $CLUSTER ;
node 1 admin conninfo = ' dbname = $DBNAME1 host = $HOST1
port = $PORT1
user = slony password = slony_user_password ';
port = $PORT2 user = slony password = slony_user_password ';
init cluster ( id = 1 , comment = ' Customers DB
replication cluster ' ) ;
echo ' Create set ';
create set ( id = 1 , origin = 1 , comment = ' Customers
DB replication set ' ) ;
echo ' Adding tables to the subscription set ';
echo ' Adding table public . customers_sales ... ';
set add table ( set id = 1 , origin = 1 , id = 4 , full qualified
name = ' public . customers_sales ' , comment = ' Table
public . customers_sales ' ) ;
echo ' done ';
45
4.2.
37
38
39
40
41
43
44
45
46
47
48
49
50
52
53
54
Slony-I
echo ' Adding table public . customers_something ... ';

set add table ( set id = 1 , origin = 1 , id = 5 , full qualified
name = ' public . customers_something ,
comment = ' Table public . customers_something ) ;
echo ' done ';
echo ' done adding ';
store node ( id = 2 , comment = ' Node 2 , $HOST2 ' ) ;
echo ' stored node ';
store path ( server = 1 , client = 2 , conninfo =
' dbname = $DBNAME1 host = $HOST1
port = $PORT1 user = slony password = slony_user_password ' ) ;
echo ' stored path ';
' dbname = $DBNAME2 host = $HOST2
port = $PORT2 user = slony password = slony_user_password ' ) ;
store listen ( origin = 1 , provider = 1 , receiver = 2 ) ;
EOF
, ,
. : ,
, id , primary key.
: replication set .
set.
: . unsubscribe subscribe .
slave- replication set

:
Listing 4.10: slave- replication set
1
3
5
6
8
9
11
12
14
# !/ bin / sh
DBNAME1 = customers
DBNAME2 = customers
HOST2 = customers_slave . com
PORT1 =5432
PORT2 =5432
SLONY_USER = slony
46
4.2.
16
17
18
19
20
21
23
24
26
Slony-I
slonik << EOF

echo ' subscribing ';
subscribe set ( id = 1 , provider = 1 , receiver = 2 , forward =
no ) ;
EOF

, .
Listing 4.11:
1
2
slony@customers_master$ slon customers_rep \

" dbname = customers user = slony "
Listing 4.12:
1
2
slony@customers_slave$ slon customers_rep \

. COPY, slave DB
.
slave-
10- . slon
, .

4.2.1 4.2.2.
id = 3. customers_slave3.com,
- PgSQL.
( 4.2.2) :
Listing 4.13:
1
2
3
slonik << EOF

cluster name = customers_slave ;
node 3 admin conninfo = ' dbname = customers
host = customers_slave3 . com
47
4.2.
4
5
6
7
Slony-I
port =5432 user = slony password = slony_user_pass ';

uninstall node ( id = 3) ;
echo ' okay ';
EOF
, ,
.
. :
Listing 4.14:
1
3
5
6
8
9
11
12
14
16
17
18
19
20
21
23
25
26
27
28
29
30
31
33
34
35
37
# !/ bin / sh
DBNAME1 = customers
DBNAME3 = customers
HOST3 = customers_slave3 . com
PORT1 =5432
PORT2 =5432
SLONY_USER = slony
slonik << EOF
port = $PORT1 user = slony password = slony_user_pass ';
node 3 admin conninfo = ' dbname = $DBNAME3
host = $HOST3 port = $PORT2 user = slony password = slony_user_pass ';
echo ' done adding ';
store node ( id = 3 , comment = ' Node 3 , $HOST3 ' ) ;
echo ' sored node ';
' dbname = $DBNAME1
host = $HOST1 port = $PORT1 user = slony password = slony_user_pass ' ) ;
echo ' stored path ';
' dbname = $DBNAME3
host = $HOST3 port = $PORT2 user = slony password = slony_user_pass ' ) ;
echo ' again ';
EOF
48
4.2.
Slony-I
id 3, 2 .
3 replication set:
Listing 4.15:
1
3
5
6
8
9
11
12
14
16
17
18
19
20
21
23
24
26
# !/ bin / sh
DBNAME1 = customers
DBNAME3 = customers
HOST3 = customers_slave3 . com
PORT1 =5432
PORT2 =5432
SLONY_USER = slony
slonik << EOF
echo ' subscribing ';
subscribe set ( id = 1 , provider = 1 , receiver = 3 , forward =
no ) ;
EOF
slon , . slon .
Listing 4.16:
1
2
slony@customers_slave3$ slon customers_rep \

.

, : , :
Listing 4.17:
49
4.2.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
Slony-I
% slon customers_rep " dbname = customers user = slony_user "

CONFIG main : slon version 1.0.5 starting up
CONFIG main : local node id = 3
CONFIG main : loading current cluster configuration
CONFIG storeNode : no_id =1 no_comment = ' CustomersDB
replication cluster '
CONFIG storeNode : no_id =2 no_comment = ' Node 2 ,
node2 . example . com '
CONFIG storeNode : no_id =4 no_comment = ' Node 4 ,
node4 . example . com '
CONFIG storePath : pa_server =1 pa_client =3
pa_conninfo = " dbname = customers
host = mainhost . com port =5432 user = slony_user
password = slony_user_pass " pa_connretry =10
CONFIG storeListen : li_origin =1 li_receiver =3
li_provider =1
CONFIG storeSet : set_id =1 set_origin =1
set_comment = ' CustomersDB replication set '
WARN remoteWorker_wakeup : node 1 - no worker thread
CONFIG storeSubscribe : sub_set =1 sub_provider =1 sub_forward = 'f '
CONFIG enableSubscription : sub_set =1
CONFIG main : configuration complete - starting threads
CONFIG enableNode : no_id =1
ERROR remoteWorkerThread_1 : " begin transaction ; set
transaction isolation level
serializable ; lock table " _customers_rep " . sl_config_lock ;
select
" _customers_rep " . enableSubscription (1 , 1 , 4) ;
notify " _customers_rep_Event " ; notify " _customers_rep_Confirm " ;
insert into " _customers_rep " . sl_event ( ev_origin , ev_seqno ,
ev_timestamp , ev_minxid , ev_maxxid , ev_xip ,
ev_type , ev_data1 , ev_data2 , ev_data3 , ev_data4 ) values
( '1 ' , '219440 ' ,
'2005 -05 -05 18:52:42.708351 ' , '52501283 ' , '52501292 ' ,
' ' '52501283 ' ' ' , ' ENABLE_SUBSCRIPTION ' ,
'1 ' , '1 ' , '4 ' , 'f ') ; insert into " _customers_rep " .
sl_confirm ( con_origin , con_received ,
con_seqno , con_timestamp ) values (1 , 3 , '219440 ' ,
CURRENT_TIMESTAMP ) ; commit transaction ; "
PGRES_FATAL_ERROR ERROR : insert or update on table
" sl_subscribe " violates foreign key
constraint " sl_subscribe - sl_path - ref "
DETAIL : Key ( sub_provider , sub_receiver ) =(1 ,4)
is not present in table " sl_path " .
INFO remoteListenThread_1 : disconnecting from
' dbname = customers host = mainhost . com
port =5432 user = slony_user password = slony_user_pass '
%
50
4.2.
Slony-I
_< >.sl_path;, _customers_rep.sl_path . , id 4, (1,4)

sl_path .
, Slony.
.
, ( ,
(1,4)):
Listing 4.18:
1
2
3
4
slony_user@masterhost$ psql -d customers -h

_every_one_of_slaves -U slony
customers = # insert into _customers_rep . sl_path
values ( '1 ' , '4 ' , ' dbname = customers host = mainhost . com
port =5432 user = slony_user password = slony_user_password , '10 ') ;
,
. _< >,
_customers_rep.

master-, SELECT-
. pg_stat_activity :
Listing 4.19:
1
2
3
4
5
6
7
8
9
10
11
12
13
select ev_origin , ev_seqno , ev_timestamp , ev_minxid ,

ev_maxxid , ev_xip ,
ev_type , ev_data1 , ev_data2 , ev_data3 , ev_data4 , ev_data5 ,
ev_data6 ,
ev_data7 , ev_data8 from " _customers_rep " . sl_event e where
( e . ev_origin = '2 ' and e . ev_seqno > '336996 ') or
( e . ev_origin = '10 ' and e . ev_seqno > '1692318 ')
order by e . ev_origin , e . ev_seqno ;
_customers_rep ,
.
51
4.3.
Londiste
sl_event - , .
:
Listing 4.20:
1
2
delete from _customers_rep . sl_event where

ev_timestamp < NOW () - '1 DAY ':: interval ;
. _customers_rep.sl_log_* , - , _customers_rep.sl_log_1 .
4.3 Londiste
Londiste , python. :
. - , Slony-I. Londiste PgQ (
,
, PostgreSQL). :
, (failover)
(switchover) ( 3

10
, Linux,
Ubuntu Server. , ( Windows) ,
10
http://skytools.projects.postgresql.org/skytools-3.0/doc/skytools3.html
52
4.3.
Londiste
PostgreSQL Windows, , .
Londiste Skytools,
. , Debian Ubuntu skytools
:
Listing 4.21:
sudo aptitude install skytools

http://pgfoundry.org/projects/skytools.
2.1.11. , :
Listing 4.22:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
$wget http :// pgfoundry . org / frs / download . php /2561/

skytools -2.1.11. tar . gz
$tar zxvf skytools -2.1.11. tar . gz
$cd skytools -2.1.11/
# deb
$sudo aptitude install build - essential autoconf \
automake autotools - dev dh - make \
debhelper devscripts fakeroot xutils lintian pbuilder \
python - dev yada
# postgresql 8.4. x
$sudo aptitude install postgresql - server - dev -8.4
# python - psycopg Londiste
$sudo aptitude install python - psycopg2
# deb
# postgresql 8.4. x ( 8.3. x " make deb83 ")
$sudo make deb84
$cd ../
# skytools
$dpkg -i skytools - modules -8.4 _2 .1.11 _i386 . deb
skytools_2 .1.11 _i386 . deb
Skytools
Listing 4.23:
1
2
3
./ configure
make
make install
,
Listing 4.24:
1
2
3
4
$londiste . py -V
Skytools version 2.1.11
$pgqadm . py -V
Skytools version 2.1.11
53
4.3.
Londiste
, .
host1 ;
host2 ;
ticker-
Londiste ticker ,
. , , , . ticker- ( /etc/skytools/db1ticker.ini):
Listing 4.25: ticker-
1
2
3
5
6
8
9
10
12
13
14
16
17
18
[ pgqadm ]
#
job_name = db1 - ticker
#
db = dbname = P host = host1
#
# ( ..)
maint_delay = 600
#
# ( )
loop_delay = 0.1
# log pid
logfile = / var / log /%( job_name ) s . log
pidfile = / var / pid /%( job_name ) s . pid
(SQL)
ticker .
pgqadm.py :
1
2
pgqadm . py / etc / skytools / db1 - ticker . ini install

pgqadm . py / etc / skytools / db1 - ticker . ini ticker -d
, (/var/log/skytools/db1-tickers.log) . ( ).
54
4.3.
Londiste
ticker,
:
pgqadm . py / etc / skytools / db1 - ticker . ini ticker -s

ticker:
pgqadm . py / etc / skytools / db1 - ticker . ini ticker -k

Londiste . slave
, .

( /etc/skytools/db1-londiste.ini):
Listing 4.29:
1
2
3
5
6
7
8
10
11
12
13
14
15
17
18
19
21
22
23
[ londiste ]
#
job_name = db1 - londiste
#
provider_db = dbname = db1 port =5432 host = host1
#
subscriber_db = dbname = db1 host = host2
#
# SQL - , ..
# .
# ! ,
# pgq_queue_name = db1 - londiste - queue
# log pid
logfile = / var / log /%( job_name ) s . log
pidfile = / var / run /%( job_name ) s . pid
#
log_size = 5242880
log_count = 3
55
4.3.
Londiste
Londiste
SQL .
:
Listing 4.30: Londiste
londiste . py / etc / skytools / db1 - londiste . ini provider install

:
Listing 4.31: Londiste
londiste . py / etc / skytools / db1 - londiste . ini subscriber install

.
Londiste
:
Listing 4.32:
londiste . py / etc / skytools / db1 - londiste . ini replay -d

, , ..
,
.
(/var/log/db1-londistes.log).

:
Listing 4.33:
londiste . py / etc / skytools / db1 - londiste . ini provider add -- all

:
Listing 4.34:
londiste . py / etc / skytools / db1 - londiste . ini subscriber add -- all

- all,
,
, .
56
4.3.
Londiste
(sequence)
. :
Listing 4.35:
londiste . py / etc / skytools / db1 - londiste . ini provider add - seq

-- all
:
Listing 4.36:
londiste . py / etc / skytools / db1 - londiste . ini subscriber add - seq

-- all
all.
, . Londiste
bulk copy , ( COPY) ,
.
:
Listing 4.37:
less / var / log / db1 - londiste . log

, . , "ok".
Listing 4.38:
1
3
4
5
6
7
8
9
10
londiste . py / etc / skytools / db1 - londiste . ini subscriber tables

Table State
public . table1
public . table2
public . table3
public . table4
public . table5
public . table6
...
ok
ok
in - copy
-

( ):
Listing 4.39:
(
57
4.3.
2
3
4
5
6
Londiste
while [ $ (
python londiste . py / etc / skytools / db1 - londiste . ini subscriber
tables |
tail -n +2 | awk '{ print $2 } ' | grep -v ok | wc -l ) - ne 0 ];
do sleep 60; done ; echo '' | mail -s ' Replication done EOM '
user@domain . com
) &

:
Listing 4.40:
londiste . py <ini > provider tables | xargs londiste . py <ini >

subscriber add

.
Listing 4.41:
1
2
SELECT queue_name , consumer_name , lag , last_seen

FROM pgq . get_consumer_info () ;
lag , last_seen
.
, 60 .

Londiste
, . PGQ, , API:
Listing 4.42:
SELECT pgq . unregister_consumer ( ' queue_name ' , ' consumer_name ') ;

pgqadm.py:
Listing 4.43:
pgqadm . py < ticker . ini > unregister queue_name consumer_name
58
4.3.
Londiste

:
1.
2. BEGIN;
3.
4. SELECT londiste.provider_refresh_trigger('queue_name', 'tablename');
5. COMMIT;

1. BEGIN;
2.
3. SELECT londiste.provider_refresh_trigger('queue_name', 'tablename');
4. COMMIT;
5. lag, londiste
6.
, ,
.
Londiste lag
, , ticker. UPDATE DELETE
,
. . .
,
pgq.subscription sub_last_tick sub_next_tick.
Listing 4.44:
1
2
3
4
5
6
7
SELECT count (*)

FROM pgq . event_1 ,
( SELECT tick_snapshot
FROM pgq . tick
WHERE tick_id BETWEEN 5715138 AND 5715139
) as t ( snapshots )
WHERE txid_visible_in_snapshot ( ev_txid , snapshots ) ;
, 5 400 . .
Londiste, . Londiste . INI
ticker- :
59
4.4.
Streaming Replication ( )
Listing 4.45:
pgq_lazy_fetch = 500
Londiste 500 . .
4.4 Streaming Replication (

)
(Streaming Replication, SR)

wall xlog
. PostgreSQL 9 ( !).
, , ,
, PostgreSQL.
:
PostgreSQL

-

PostgreSQL
(
, ,
,
, Hot Standby)
PostgreSQL 9 .
9.0.1 . , , Linux.
masterdb(192.168.0.10)
slavedb(192.168.0.20).
60
4.4.

ssh. postgres . ,
:
Listing 4.46: userssh
1
2
3
$sudo groupadd userssh

$sudo useradd -m -g userssh -d / home / userssh -s / bin / bash \
-c " user ssh allow " userssh
(
postgres):
Listing 4.47: postgres
su postgres
RSA- :
Listing 4.48: RSA-
1
2
3
4
5
6
7
8
postgres@localhost ~ $ ssh - keygen -t rsa -P " "

Generating public / private rsa key pair .
Enter file in which to save the key
(/ var / lib / postgresql /. ssh / id_rsa ) :
Created directory '/ var / lib / postgresql /. ssh '.
Your identification has been saved in
/ var / lib / postgresql /. ssh / id_rsa .
Your public key has been saved in
/ var / lib / postgresql /. ssh / id_rsa . pub .
The key fingerprint is :
16:08:27:97:21:39: b5 :7 b :86: e1 :46:97: bf :12:3 d :76
postgres@localhost
:
Listing 4.49:
cat $HOME /. ssh / id_rsa . pub >> $HOME /. ssh / authorized_keys

, :
Listing 4.50: ssh
ssh localhost
sshd:
Listing 4.51: sshd
/ etc / init . d / sshd start

61
4.4.
$HOME/.ssh slavedb.

ssh.
pg_hba.conf ,
(trust):
Listing 4.52: pg_hba.conf
host
replication
all
192.168.0.20/32
trust
Listing 4.53: pg_hba.conf
host
replication
all
192.168.0.10/32
trust
postgresql .

masterdb. postgresql.conf
:
Listing 4.54:
1
2
3
4
6
7
9
10
11
12
13
14
15
16
18
# To enable read - only queries on a standby server , wal_level

must be set to
# " hot_standby ". But you can choose " archive " if you never
connect to the
# server in standby mode .
wal_level = hot_standby
# Set the maximum number of concurrent connections from the
standby servers .
max_wal_senders = 5
# To prevent the primary server from removing the WAL segments
required for
# the standby server before shipping them , set the minimum
number of segments
# retained in the pg_xlog directory . At least
wal_keep_segments should be
# larger than the number of segments generated between the
beginning of
# online - backup and the startup of streaming replication . If
you enable WAL
# archiving to an archive directory accessible from the
standby , this may
# not be necessary .
wal_keep_segments = 32
# Enable WAL archiving on the primary to an archive directory
accessible from
62
4.4.
19
20
21
22
# the standby . If wal_keep_segments is a high enough number to

retain the WAL
# segments required for the standby server , this may not be
necessary .
archive_mode
= on
archive_command = ' cp % p / path_to / archive /% f '
:
wal_level = hot_standby WAL

archive, ,
( archive,
).
max_wal_senders = 5 .
wal_keep_segments = 32 c WAL
pg_xlog .
archive_mode = on WAL archive_command .

/path_to/archive/.
postgresql .
slavedb.

slavedb masterdb.
.
masterdb . :
Listing 4.55:
psql -c " SELECT pg_start_backup ( ' label ' , true ) "

.
:
Listing 4.56:
1
2
3
rsync -C -a -- delete -e ssh -- exclude postgresql . conf

-- exclude postmaster . pid \
-- exclude postmaster . opts -- exclude pg_log -- exclude pg_xlog \
-- exclude recovery . conf master_db_datadir /
slavedb_host : slave_db_datadir /
master_db_datadir postgresql masterdb

slave_db_datadir postgresql slavedb
slavedb_host slavedb( - 192.168.1.20)
63
4.4.
, . :
Listing 4.57:
psql -c " SELECT pg_stop_backup () "

postgresql.conf, ( ). :
Listing 4.58:
hot_standby = on
! wal_level = archive, (hot_standby = o ).
slavedb PostgreSQL
recovery.conf :
Listing 4.59: recovery.conf
1
2
3
5
6
7
9
10
11
13
14
15
16
17
18
# Specifies whether to start the server as a standby . In

streaming replication ,
# this parameter must to be set to on .
standby_mode
= 'on '
# Specifies a connection string which is used for the standby
server to connect
# with the primary .
primary_conninfo
= ' host =192.168.0.10 port =5432
user = postgres '
# Specifies a trigger file whose presence should cause
streaming replication to
# end ( i . e . , failover ) .
trigger_file = '/ path_to / trigger '
# Specifies a command to load archive segments from the WAL
archive . If
# wal_keep_segments is a high enough number to retain the WAL
segments
# required for the standby server , this may not be necessary .
But
# a large workload can cause segments to be recycled before
the standby
# is fully synchronized , requiring you to start again from a
new base backup .
restore_command = ' scp masterdb_host :/ path_to / archive /% f " % p " '
standby_mode='on'
64
4.4.
primary_conninfo
trigger_le -, .
restore_command , WAL
. scp masterdb (masterdb_host
- masterdb).
PostgreSQL slavedb.

:
Listing 4.60:
1
2
3
4
5
7
8
9
10
11
13
14
15
16
17
$ psql -c " SELECT pg_current_xlog_location () " - h192 .168.0.10

( masterdb )
pg_current_xlog_location
-------------------------0/2000000
(1 row )
$ psql -c " select pg_last_xlog_receive_location () "
- h192 .168.0.20 ( slavedb )
pg_last_xlog_receive_location
------------------------------0/2000000
(1 row )
$ psql -c " select pg_last_xlog_replay_location () "
- h192 .168.0.20 ( slavedb )
pg_last_xlog_replay_location
-----------------------------0/2000000
(1 row )
ps:
Listing 4.61:
1
2
4
5
[ masterdb ] $ ps - ef | grep sender

postgres 6879 6831 0 10:31 ?
00:00:00 postgres : wal
sender process postgres 127.0.0.1(44663) streaming 0/2000000
[ slavedb ] $ ps - ef | grep receiver
postgres 6878 6872 1 10:31 ?
00:00:01 postgres : wal
receiver process
streaming 0/2000000
. :
Listing 4.62:
65
4.4.
1
2
3
4
5
6
7
$psql test_db
test_db =# create table test3 ( id int not null primary key , name
varchar (20) ) ;
NOTICE : CREATE TABLE / PRIMARY KEY will create implicit index
" test3_pkey " for table " test3 "
CREATE TABLE
test_db =# insert into test3 ( id , name ) values ( '1 ' , ' test1 ') ;
INSERT 0 1
test_db =#
:
Listing 4.63:
1
2
3
4
5
6
$psql test_db
test_db =# select * from test3 ;
id | name
-- - -+ - - - - - - 1 | test1
(1 row )
,
.

- (trigger_le) , .

- (trigger_le) .

. , .

PostgreSQL .

, ,
. PostgreSQL .
66
4.5.
Bucardo
4.5 Bucardo
Bucardo master-master master-slave PostgreSQL,

Perl. ,
.
Ubuntu Server. DBIx::Safe Perl .

Listing 4.64:
sudo aptitude install libdbix - safe - perl

11
Listing 4.65:
1
2
3
4
tar xvfz DBIx - Safe -1.2.5. tar . gz

cd DBIx - Safe -1.2.5
perl Makefile . PL
make && make test && sudo make install
Bucardo.
12
Listing 4.66:
1
2
3
4
5
tar xvfz Bucardo -4.4.0. tar . gz

cd Bucardo -4.4.0
perl Makefile . PL
make
sudo make install
Bucardo pl/perlu
PostgreSQL.
Listing 4.67:
sudo aptitude install postgresql - plperl -8.4

.
Bucardo
:
11
12
http://search.cpan.org/CPAN/authors/id/T/TU/TURNSTEP/
http://bucardo.org/wiki/Bucardo#Obtaining_Bucardo
67
4.5.
Bucardo
Listing 4.68: Bucardo
bucardo_ctl install
Bucardo PostgreSQL, :
Listing 4.69: Bucardo
1
2
3
5
6
8
9
10
11
12
13
This will install the bucardo database into an existing

Postgres cluster .
Postgres must have been compiled with Perl support ,
and you must connect as a superuser
We will create a new superuser named ' bucardo ' ,
and make it the owner of a new database named ' bucardo '
Current connection settings :
1. Host :
< none >
2. Port :
5432
3. User :
postgres
4. Database :
postgres
5. PID directory : / var / run / bucardo
, Bucardo
bucardo bucardo. Unix socket, pg_hda.conf.

, Bucardo. master_db slave_db.
:
Listing 4.70:
1
2
3
bucardo_ctl add db master_db name = master

bucardo_ctl add all tables herd = all_tables
bucardo_ctl add all sequences herd = all_tables
master (
, master_db slave_db Bucardo ).
,
all_tables.
slave_db:
Listing 4.71:
bucardo_ctl add db slave_db name = replica port =6543

host = slave_host
68
4.5.
Bucardo
replica Bucardo.

. (master-slave):
Listing 4.72:
bucardo_ctl add sync delta type = pushdelta source = all_tables

targetdb = replica
Bucardo PostgreSQL. :
type
. 3 :
Fullcopy. .
Pushdelta. Master-slave .
Swap. Master-master .
Bucardo
. goat ( ) standard_conict
(
):
source source (master_db
).
target target (slave_db
).
skip . -
.
random , -
.
latest ,
.
abort .
source
.
targetdb
, .
master-master:
Listing 4.73:
bucardo_ctl add sync delta type = swap source = all_tables

targetdb = replica
69
4.6.
RubyRep
/
:
Listing 4.74:
bucardo_ctl start
:
Listing 4.75:
bucardo_ctl stop

:
Listing 4.76:
bucardo_ctl show all

Listing 4.77:
bucardo_ctl set name = value

:
Listing 4.78:
bucardo_ctl set syslog_facility = LOG_LOCAL3

Listing 4.79:
bucardo_ctl reload_config
http://bucardo.org/wiki/Bucardo_ctl
4.6 RubyRep
RubyRep , ruby. : . master-master,
70
4.6.
RubyRep
master-slave , PostgreSQL MySQL. :
MySQL

RubyRep : Ruby
JRuby. JRuby .
JRuby
Java ( 1.6).
13
1. JRuby rubyrep c Rubyforge

2.
3.
Ruby
1. Ruby, Rubygems.
2. .
Mysql:
Listing 4.80:
sudo gem install mysql

PostgreSQL:
Listing 4.81:
sudo gem install postgres
3. rubyrep:
Listing 4.82:
1
13
sudo gem install rubyrep
http://rubyforge.org/frs/?group_id=7932, ZIP
71
4.6.
RubyRep

:
Listing 4.83:
rubyrep generate myrubyrep . conf

generate myrubyrep.conf:
Listing 4.84:
1
2
3
4
5
6
7
8
10
11
12
13
14
15
16
18
19
20
21
RR :: Initializer :: run do | config |

config . left = {
: adapter = > ' postgresql ' , # or ' mysql '
: database = > ' SCOTT ' ,
: username = > ' scott ' ,
: password = > ' tiger ' ,
: host
= > '172.16.1.1 '
}
config . right
: adapter = >
: database = >
: username = >
: password = >
: host
=>
}
= {
' postgresql ' ,
' SCOTT ' ,
' scott ' ,
' tiger ' ,
'172.16.1.2 '
config . include_tables ' dept '

config . include_tables /^ e / # regexp matches all tables
starting with e
# config . include_tables /./ # regexp matches all tables
end
. left
right. cong.include_tables ( RegEx).

:
Listing 4.85:
rubyrep scan -c myrubyrep . conf

:
Listing 4.86:
1
2
dept 100% .........................

emp 100% .........................
72
4.6.
RubyRep
dept , emp
.

:
Listing 4.87:
rubyrep sync -c myrubyrep . conf

:
Listing 4.88:
rubyrep sync -c myrubyrep . conf dept /^ e /

. http://www.rubyrep.org/conguration.html.
:
Listing 4.89:
rubyrep replicate -c myrubyrep . conf

( ) . , . ,
rubyrep. ,
.
:
Listing 4.90:
rubyrep uninstall -c myrubyrep . conf

rubyrep Ruby :
Listing 4.91:
1
2
$rubyrep replicate -c myrubyrep . conf

Verifying RubyRep tables
73
4.7.
3
4
5
6
Checking for and removing rubyrep triggers from unconfigured

tables
Verifying rubyrep triggers of configured tables
Starting replication
Exception caught : Thread # join : deadlock 0 xb76ee1ac - mutual
join (0 xb758cfac )
Ruby. :
1. rubyrep JRuby ( )
2. rubyrep :
Listing 4.92:
1
2
3
4
6
7
8
9
10
11
12
13
15
16
17
18
19
20
21
22
23
24
25
26
27
--- / Library / Ruby / Gems /1.8/ gems / rubyrep -1.1.2/ lib / rubyrep /
replication_runner . rb 2010 -07 -16 15:17:16.000000000 -0400
+++ ./ replication_runner . rb 2010 -07 -16 17:38:03.000000000
-0400
@@ -2 ,6 +2 ,12 @@
require ' optparse '
require ' thread '
+ require ' monitor '
+
+ class Monitor
+ alias lock mon_enter
+ alias unlock mon_exit
+ end
module RR
# This class implements the functionality of the
' replicate ' command .
@@ -94 ,7 +100 ,7 @@
# Initializes the waiter thread used for replication
pauses
# and processing
# the process TERM signal .
def init_waiter
- @termination_mutex = Mutex . new
+ @termination_mutex = Monitor . new
@termination_mutex . lock
@waiter_thread ||= Thread . new
{ @termination_mutex . lock ;
self . termination_requested = true }
% w ( TERM INT ) . each do | signal |
4.7
, PostgreSQL.
74
4.7.
, , ..
PostgreSQL.
.
, 9.0 PostgreSQL. Slony-I
, , , (failover) (switchover).
Londiste ,
. Bucardo mastermaster, master-slave ,
, (failover) (switchover). RubyRep, master-master ,
, ( ).
75
,
.

5.1
. .
. , . .
,
(, 90% ). , ( !)
, ().
, , , ..
.
, , . ..
. ? . ,
. ,
10 , 10 - ..
.. - ,
, .
, ID
(user_id) -
76
5.1.
. .. :
,
user_id
- =. , ,
.
, .
=
() . , ,
( , ).
.
,
.. .
, ,
, (, ,
). . , ,
, ID , 100 .
. PostgreSQL
:
Greenplum Database1
GridSQL for EnterpriseDB Advanced Server2
Sequoia3
PL/Proxy4
HadoopDB5 (Shared-nothing clustering)
http://www.greenplum.com/index.php?page=greenplum-database
http://www.enterprisedb.com/products/gridsql.do
3 http://www.continuent.com/community/lab-projects/sequoia
4 http://plproxy.projects.postgresql.org/doc/tutorial.html
5 http://db.cs.yale.edu/hadoopdb/hadoopdb.html
1
2
77
5.2.
PL/Proxy
5.2 PL/Proxy
PL/Proxy - .
, , , (,
, , - ).
PL/Proxy ? .
, ,
26 . ,
-, : , , - . .
PL/Proxy
OLTP . failover-
, -, .
:
autocommit-

SELECT;

- ;
-,
PgBouncer
( , )
-
1. PL/Proxy
2. PL/Proxy make make install.

PL/Proxy .
Ubuntu Server PostgreSQL 8.4:
Listing 5.1:
sudo aptitude install postgresql -8.4 - plproxy

6
http://pgfoundry.org/projects/plproxy
78
5.2.
PL/Proxy
3 PostgreSQL. 2
node1 node2, ,
proxy. pl/proxy
.
plproxytest, users. !
node1 node2.
.
plproxytest( ):
Listing 5.2:
1
2
3
CREATE DATABASE plproxytest

WITH OWNER = postgres
ENCODING = ' UTF8 ';
users:
Listing 5.3:
1
2
3
4
5
6
7
CREATE TABLE public . users

(
username character varying (255) ,
email character varying (255)
)
WITH ( OIDS = FALSE ) ;
ALTER TABLE public . users OWNER TO postgres ;
users:
Listing 5.4:
1
2
3
4
5
6
7
8
9
CREATE OR REPLACE FUNCTION public . insert_user ( i_username text ,

i_emailaddress
text )
RETURNS integer AS
$BODY$
INSERT INTO public . users ( username , email ) VALUES ( $1 , $2 ) ;
SELECT 1;
$BODY$
LANGUAGE ' sql ' VOLATILE ;
ALTER FUNCTION public . insert_user ( text , text ) OWNER TO
postgres ;
. proxy.
, (proxy) :
Listing 5.5:
1
2
3
CREATE DATABASE plproxytest

WITH OWNER = postgres
ENCODING = ' UTF8 ';
79
5.2.
PL/Proxy
pl/proxy:
Listing 5.6:
1
2
3
4
5
6
7
8
9
10
CREATE OR REPLACE FUNCTION public . plproxy_call_handler ()

RETURNS language_handler AS
' $libdir / plproxy ' , ' plproxy_call_handler '
LANGUAGE 'c ' VOLATILE
COST 1;
ALTER FUNCTION public . plproxy_call_handler ()
OWNER TO postgres ;
-- language
CREATE LANGUAGE plproxy HANDLER plproxy_call_handler ;
CREATE LANGUAGE plpgsql ;
, 3 pl/proxy
. .
kay-value:
Listing 5.7:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
CREATE OR REPLACE FUNCTION public . get_cluster_config

( IN cluster_name text ,
OUT " key " text , OUT val text )
RETURNS SETOF record AS
$BODY$
BEGIN
-- lets use same config for all clusters
key := ' connection_lifetime ';
val := 30*60; -- 30 m
RETURN NEXT ;
RETURN ;
END ;
$BODY$
LANGUAGE ' plpgsql ' VOLATILE
COST 100
ROWS 1000;
ALTER FUNCTION public . get_cluster_config ( text )
OWNER TO postgres ;
.
DSN :
Listing 5.8:
1
2
3
4
5
6
7
CREATE OR REPLACE FUNCTION

public . get_cluster_partitions ( cluster_name text )
RETURNS SETOF text AS
$BODY$
BEGIN
IF cluster_name = ' usercluster ' THEN
RETURN NEXT ' dbname = plproxytest host = node1 user = postgres ';
80
5.2.
8
9
10
11
12
13
14
15
16
17
18
PL/Proxy
RETURN NEXT ' dbname = plproxytest host = node2 user = postgres ';
RETURN ;
END IF ;
RAISE EXCEPTION ' Unknown cluster ';
END ;
$BODY$
COST 100
ROWS 1000;
ALTER FUNCTION public . get_cluster_partitions ( text )
OWNER TO postgres ;
:
Listing 5.9:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

public . get_cluster_version ( cluster_name text )
RETURNS integer AS
$BODY$
BEGIN
IF cluster_name = ' usercluster ' THEN
RETURN 1;
END IF ;
RAISE EXCEPTION ' Unknown cluster ';
END ;
$BODY$
COST 100;
ALTER FUNCTION public . get_cluster_version ( text )
OWNER TO postgres ;

:
Listing 5.10:
1
2
3
4
5
6
7
8
9
10
11

public . insert_user ( i_username text , i_emailaddress text )
RETURNS integer AS
$BODY$
CLUSTER ' usercluster ';
RUN ON hashtext ( i_username ) ;
$BODY$
LANGUAGE ' plproxy ' VOLATILE
COST 100;
ALTER FUNCTION public . insert_user ( text , text )
OWNER TO postgres ;
. proxy :
Listing 5.11:
SELECT insert_user ( ' Sven ' , ' sven@somewhere . com ') ;

81
5.2.
2
3
PL/Proxy
SELECT insert_user ( ' Marko ' , ' marko@somewhere . com ') ;

SELECT insert_user ( ' Steve ' , ' steve@somewhere . com ') ;
. :
Listing 5.12:
1
2
3
4
5
6
7
8
9
10
11
12
13
14

public . get_user_email ( i_username text )
RETURNS SETOF text AS
$BODY$
CLUSTER ' usercluster ';
RUN ON hashtext ( i_username ) ;
SELECT email FROM public . users
WHERE username = i_username ;
$BODY$
LANGUAGE ' plproxy ' VOLATILE
COST 100
ROWS 1000;
ALTER FUNCTION public . get_user_email ( text )
OWNER TO postgres ;
:
Listing 5.13:
select plproxy . get_user_email ( ' Steve ') ;

,
, users .
?
pl/proxy
. ,
. 16 .
- . ?
Highload++ 2008,

Skype,
opensource.
,
. ,
,
get_cluster_partitions.
82
5.3.
HadoopDB
5.3 HadoopDB
Hadoop ,
. ,
:
Hadoop -
, ;
: , .
;
,
;
, .

;
, Java,
, JVM.
HDFS

Hadoop Distributed File System.
,
:
, , ;

;
;
;
:
;
: , ;
HDFS
Namenode
.
.
83
5.3.
HadoopDB
. 5.1: HDFS

, . .
Datanode
.

. Rack, , , .
, . .
HDFS : .
( , ; - 64 mb) , Datanode'.
,
. , -
,
84
5.3.
HadoopDB
/trash
. ,
.
- , Datanode
Namenode' . Namenode
, - . ,

.

, TCP/IP.
Namenode ClientProtocol,
DatanodeProtocol,
Remote Procedure Call (RPC).
, DFSShell, DFSAdmin,
, -. API : Java API, C pipeline,
WebDAV .
MapReduce
, Hadoop framework
,
. Job ()
, , :
Map
(
-) -.
.
Reduce
map .
, .
, ,
( ).
JobTracker, TaskTracker. JobTracker' , ,
.
85
5.3.
HadoopDB
, framework',
map reduce,
.
JobTracker .

, , , .
HBase
Hadoop ,
.
Google
BigTable.
HBase
. , .
HQL ( Hadoop Query
Language), SQL.
.
HBase
, --, .
,
, , . , .
,
.
HQL , SQL,
help;, .
SELECT, INSERT, UPDATE, DROP , .
HBase Shell, HBase
API : Java, Jython, REST Thrift.
HadoopDB
HadoopDB Yale Brown , MapReduce, .

MapReduce , , -
86
5.3.
HadoopDB
. SQL,
MapReduce, . MapReduce ,
.

Ubuntu Server .
Hadoop
, Hadoop,
, :
ssh ,
[hadoop]:
Listing 5.14:
1
2
3
$sudo groupadd hadoop

$sudo useradd -m -g hadoop -d / home / hadoop -s / bin / bash \
-c " Hadoop software owner " hadoop
:
Listing 5.15: hadoop
su hadoop
RSA- :
Listing 5.16: RSA-
1
2
3
4
5
6
7
hadoop@localhost ~ $ ssh - keygen -t rsa -P " "

Generating public / private rsa key pair .
Enter file in which to save the key
(/ home / hadoop /. ssh / id_rsa ) :
Your identification has been saved in
/ home / hadoop /. ssh / id_rsa .
Your public key has been saved in
/ home / hadoop /. ssh / id_rsa . pub .
The key fingerprint is :
7 b :5 c : cf :79:6 b :93: d6 : d6 :8 d :41: e3 : a6 :9 d :04: f9 :85
hadoop@localhost
87
5.3.
HadoopDB
:
Listing 5.17:
cat $HOME /. ssh / id_rsa . pub >> $HOME /. ssh / authorized_keys

, :
Listing 5.18: ssh
ssh localhost
sshd:
Listing 5.19: sshd
/ etc / init . d / sshd start

JVM
1.5.0 .
Listing 5.20: JVM
sudo aptitude install openjdk -6 - jdk
Hadoop:
Listing 5.21: Hadoop
1
2
3
4
5
6
7
8
cd / opt
sudo wget http :// www . gtlib . gatech . edu / pub / apache / hadoop
/ core / hadoop -0.20.2/ hadoop -0.20.2. tar . gz
sudo tar zxvf hadoop -0.20.2. tar . gz
sudo ln -s / opt / hadoop -0.20.2 / opt / hadoop
sudo chown -R hadoop : hadoop / opt / hadoop / opt / hadoop -0.20.2
sudo mkdir -p / opt / hadoop - data / tmp - base
sudo chown -R hadoop : hadoop / opt / hadoop - data /
/opt/hadoop/conf/hadoop-env.sh :
Listing 5.22:
1
2
3
4
5
6
export
export
export
export
export
export
JAVA_HOME =/ usr / lib / jvm / java -6 - openjdk

HADOOP_HOME =/ opt / hadoop
HADOOP_CONF = $HADOOP_HOME / conf
HADOOP_PATH = $HADOOP_HOME / bin
HIVE_HOME =/ opt / hive
HIVE_PATH = $HIVE_HOME / bin
88
5.3.
HadoopDB
export PATH = $HIVE_PATH : $HADOOP_PATH : $PATH

/opt/hadoop/conf/hadoop-site.xml:
Listing 5.23: hadoop
1
2
3
4
5
6
8
9
10
11
12
13
14
16
17
18
19
20
21
22
< configuration >

< property >
< name > hadoop . tmp . dir </ name >
< value >/ opt / hadoop - data / tmp - base </ value >
< description >A base for other temporary
directories </ description >
</ property >
< property >
< name > fs . default . name </ name >
< value > localhost:54311 </ value >
< description >
The name of the default file system .
</ description >
</ property >
< property >
< name > hadoopdb . config . file </ name >
< value > HadoopDB . xml </ value >
< description > The name of the HadoopDB
cluster configuration file </ description >
</ property >
</ configuration >
/opt/hadoop/conf/mapred-site.xml:
Listing 5.24: mapreduce
1
2
3
4
5
6
7
8
9
10
< configuration >

< property >
< name > mapred . job . tracker </ name >
< value > localhost:54310 </ value >
< description >
The host and port that the
MapReduce job tracker runs at .
</ description >
</ property >
</ configuration >
/opt/hadoop/conf/hdfs-site.xml:
Listing 5.25: hdfs
1
2
3
4
< configuration >

< property >
< name > dfs . replication </ name >
< value >1 </ value >
89
5.3.
5
6
7
8
9
HadoopDB
< description >

Default block replication .
</ description >
</ property >
</ configuration >
Namenode:
Listing 5.26: Namenode
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
$ hadoop namenode - format

10/05/07 14:24:12 INFO namenode . NameNode : STARTUP_MSG :
/************************************************************
STARTUP_MSG : Starting NameNode
STARTUP_MSG :
host = hadoop1 /127.0.1.1
STARTUP_MSG :
args = [ - format ]
STARTUP_MSG :
version = 0.20.2
STARTUP_MSG :
build = https :// svn . apache . org / repos
/ asf / hadoop / common / branches / branch -0.20 -r
911707; compiled by ' chrisdo ' on Fri Feb 19 08:07:34 UTC 2010
************************************************************/
10/05/07 14:24:12 INFO namenode . FSNamesystem :
fsOwner = hadoop , hadoop
supergroup = supergroup
isPermissionEnabled = true
10/05/07 14:24:12 INFO common . Storage :
Image file of size 96 saved in 0 seconds .
10/05/07 14:24:12 INFO common . Storage :
Storage directory / opt / hadoop - data / tmp - base / dfs / name has been
successfully formatted .
10/05/07 14:24:12 INFO namenode . NameNode :
SHUTDOWN_MSG :
/************************************************************
SHUTDOWN_MSG : Shutting down NameNode at hadoop1 /127.0.1.1
************************************************************/
. Hadoop:
Listing 5.27: Hadoop
1
2
3
4
5
6
7
8
9
10
11
$ start - all . sh
starting namenode , logging to / opt / hadoop / bin /..
/ logs / hadoop - hadoop - namenode - hadoop1 . out
localhost : starting datanode , logging to
/ opt / hadoop / bin /../ logs / hadoop - hadoop - datanode - hadoop1 . out
localhost : starting secondarynamenode , logging to
/ opt / hadoop / bin /../ logs / hadoop - hadoop - secondarynamenode - hadoop1 . out
starting jobtracker , logging to
/ opt / hadoop / bin /../ logs / hadoop - hadoop - jobtracker - hadoop1 . out
localhost : starting tasktracker , logging to
/ opt / hadoop / bin /../ logs / hadoop - hadoop - tasktracker - hadoop1 . out
Hadoop stop-all.sh.
90
5.3.
HadoopDB
HadoopDB Hive
7
HaddopDB hadoopdb.jar $HADOOP_HOME/lib:

Listing 5.28: HadoopDB
$cp hadoopdb . jar $HADOOP_HOME / lib

8
PostgreSQL JDBC .
$HADOOP_HOME/lib.
Hive HadoopDB SQL . HDFS Hive:
1
2
3
4
hadoop
hadoop
hadoop
hadoop
fs
fs
fs
fs
- mkdir
- mkdir
- chmod
- chmod
/ tmp
/ user / hive / warehouse
g + w / tmp
g + w / user / hive / warehouse
HadoopDB SMS_dist. :
1
2
3
tar zxvf SMS_dist . tar . gz

sudo mv dist / opt / hive
sudo chown -R hadoop : hadoop hive
Hadoop,
Hive :
1
2
3
4
6
$ hive
Hive history file =/ tmp / hadoop /
hive_job_log_hadoop_201005081717_1990651345 . txt
hive >
hive > quit ;
. :
Listing 5.32:
1
2
svn co http :// graffiti . cs . brown . edu / svn / benchmarks /

cd benchmarks / datagen / teragen
benchmarks/datagen/teragen/teragen.pl :
7
8
http://sourceforge.net/projects/hadoopdb/les/
http://jdbc.postgresql.org/download.html
91
5.3.
HadoopDB
Listing 5.33:
1
2
4
5
7
8
9
10
11
12
13
15
16
17
18
19
20
21
22
23
25
26
27
28
29
30
31
32
33
34
35
36
37
38
use strict ;
use warnings ;
my $CUR_HOSTNAME = ` hostname -s `;
chomp ( $CUR_HOSTNAME ) ;
my
my
my
my
my
my
my
$NUM_OF_RECORDS_1TB
= 10000000000;
$NUM_OF_RECORDS_535MB = 100;
$BASE_OUTPUT_DIR
= " / data " ;
$PATTERN_STRING
= " XYZ " ;
$PATTERN_FREQUENCY = 108299;
$TERAGEN_JAR
= " teragen . jar " ;
$HADOOP_COMMAND
= $ENV { ' HADOOP_HOME ' }. " / bin / hadoop " ;
my % files = ( " 535 MB " = > 1 ,

);
system ( " $HADOOP_COMMAND fs - rmr $BASE_OUTPUT_DIR " ) ;
foreach my $target ( keys % files ) {
my $output_dir = $BASE_OUTPUT_DIR . " / SortGrep$target " ;
my $num_of_maps = $files { $target };
my $num_of_records = ( $target eq " 535 MB " ?
$NUM_OF_RECORDS_535MB : $NUM_OF_RECORDS_1TB ) ;
print " Generating $num_of_maps files in ' $output_dir '\ n " ;
##
## EXEC : hadoop jar teragen . jar 10000000000
## / data / SortGrep / XYZ 108299 100
##
my @args = ( $num_of_records ,
$output_dir ,
$PATTERN_STRING ,
$PATTERN_FREQUENCY ,
$num_of_maps ) ;
my $cmd = " $HADOOP_COMMAND jar $TERAGEN_JAR " . join ( " " , @args ) ;
print " $cmd \ n " ;
system ( $cmd ) == 0 || die ( " ERROR : $ ! " ) ;
} # FOR
exit (0) ;
Perl ,
HDFS.
, HDFS. .
, ,
HDFS, :
Listing 5.34:
1
2
3
$hadoop fs - get / data / SortGrep535MB / part -00000 my_file

$psql
psql > CREATE DATABASE grep0 ;
92
5.3.
4
5
6
7
8
9
HadoopDB
psql > USE grep0 ;

psql > CREATE TABLE grep (
->
key1 character varying (255) ,
->
field character varying (255)
-> ) ;
COPY grep FROM ' my_file ' WITH DELIMITER '| ';
HadoopDB. HadoopDB
Catalog.properties. :
Listing 5.35:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
# Properties for Catalog Generation

# #################################
nodes_file = machines . txt
relations_unchunked = grep , EntireRankings
relations_chunked = Rankings , UserVisits
catalog_file = HadoopDB . xml
##
# DB Connection Parameters
##
port =5432
username = postgres
password = password
driver = com . postgresql . Driver
url_prefix = jdbc \: postgresql \://
##
# Chunking properties
##
chunks_per_node =0
unchunked_db_prefix = grep
chunked_db_prefix = cdb
##
# Replication Properties
##
dump_script_prefix =/ root / dump_
replication_script_prefix =/ root / load_replica_
dump_file_u_prefix =/ mnt / dump_udb
dump_file_c_prefix =/ mnt / dump_cdb
##
# Cluster Connection
##
ssh_key = id_rsa
machines.txt localhost (
). HadoopDB HDFS:
Listing 5.36:
1
2
3
4
java - cp $HADOOP_HOME / lib / hadoopdb . jar \

> edu . yale . cs . hadoopdb . catalog . SimpleCatalogGenerator \
> Catalog . properties
hadoop dfs - put HadoopDB . xml HadoopDB . xml
93
5.3.
HadoopDB
:
Listing 5.37:
java - cp hadoopdb . jar

edu . yale . cs . hadoopdb . catalog . SimpleRandomReplicationFactorTwo
Catalog . properties
HadoopDB.xml, .
HDFS:
Listing 5.38:
1
2
hadoop dfs - rmr HadoopDB . xml

hadoop dfs - put HadoopDB . xml HadoopDB . xml
true hadoopdb.cong.replication HADOOP_HOME/conf/hadoopsite.xml.
HadoopDB. , HDFS:
Listing 5.39:
1
2
3
java - cp $CLASSPATH : hadoopdb . jar \

> edu . yale . cs . hadoopdb . benchmark . GrepTaskDB \
> - pattern % wo % - output padraig - hadoop . config . file
HadoopDB . xml
:
Listing 5.40:
1
2
3
4
5
6
7
8
9
10
11
12
$java - cp $CLASSPATH : hadoopdb . jar

edu . yale . cs . hadoopdb . benchmark . GrepTaskDB \
> - pattern % wo % - output padraig - hadoop . config . file
HadoopDB . xml
14.08.2010 19:08:48 edu . yale . cs . hadoopdb . exec . DBJobBase
initConf
INFO : SELECT key1 , field FROM grep WHERE field LIKE '%% wo %% ';
14.08.2010 19:08:48 org . apache . hadoop . metrics . jvm . JvmMetrics
init
INFO : Initializing JVM Metrics with processName = JobTracker ,
sessionId =
14.08.2010 19:08:48 org . apache . hadoop . mapred . JobClient
configureCommandLineOptions
WARNING : Use GenericOptionsParser for parsing the arguments .
Applications should implement Tool for the same .
monitorAndPrintJob
INFO : Running job : job_local_0001
14.08.2010 19:08:48
edu . yale . cs . hadoopdb . connector . AbstractDBRecordReader
getConnection
94
5.3.
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
HadoopDB
INFO : Data locality failed for leo - pgsql

14.08.2010 19:08:48
getConnection
INFO : Task from leo - pgsql is connecting to chunk 0 on host
localhost with
db url jdbc : postgresql :// localhost :5434/ grep0
14.08.2010 19:08:48 org . apache . hadoop . mapred . MapTask
runOldMapper
INFO : numReduceTasks : 0
14.08.2010 19:08:48
edu . yale . cs . hadoopdb . connector . AbstractDBRecordReader close
INFO : DB times ( ms ) : connection = 104 , query execution = 20 ,
row retrieval = 79
14.08.2010 19:08:48
INFO : Rows retrieved = 3
14.08.2010 19:08:48 org . apache . hadoop . mapred . Task done
INFO : Task : attempt_local_0001_m_000000_0 is done . And is in
the process of commiting
14.08.2010 19:08:48
org . apache . hadoop . mapred . LocalJobRunner$Job statusUpdate
INFO :
14.08.2010 19:08:48 org . apache . hadoop . mapred . Task commit
INFO : Task attempt_local_0001_m_000000_0 is allowed to commit
now
14.08.2010 19:08:48
org . apache . hadoop . mapred . FileOutputCommitter commitTask
INFO : Saved output of task ' attempt_local_0001_m_000000_0 ' to
file :/ home / leo / padraig
14.08.2010 19:08:48
INFO :
14.08.2010 19:08:48 org . apache . hadoop . mapred . Task sendDone
INFO : Task ' attempt_local_0001_m_000000_0 ' done .
monitorAndPrintJob
INFO : map 100% reduce 0%
monitorAndPrintJob
INFO : Job complete : job_local_0001
14.08.2010 19:08:49 org . apache . hadoop . mapred . Counters log
INFO : Counters : 6
INFO :
FileSystemCounters
INFO :
FILE_BYTES_READ =141370
INFO :
FILE_BYTES_WRITTEN =153336
INFO :
Map - Reduce Framework
INFO :
Map input records =3
95
5.3.
51
52
53
54
55
56
57
58
59
HadoopDB

INFO :
Spilled Records =0
INFO :
Map input bytes =3
INFO :
Map output records =3
14.08.2010 19:08:49 edu . yale . cs . hadoopdb . exec . DBJobBase run
INFO :
JOB TIME : 1828 ms .
HDFS, padraig:
Listing 5.41:
1
2
3
$ cd padraig
$ cat part -00000
some data
PostgreSQL:
Listing 5.42:
1
2
3
4
5
6
8
10
psql > select * from grep where field like '% wo % ';
+ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -+ - - - - - - - - - - - - - - - - - - -+
| key1
| field
|
+ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -+ - - - - - - - - - - - - - - - - - - -+
some data
1 rows in set (0.00 sec )
psql >
. .
. PostgreSQL:
Listing 5.43:
1
2
3
4
psql > INSERT into grep ( key1 ,

' Maybe ') ;
' Maybewqe ') ;
' Maybewqesad ') ;
string ! ') ;
field ) VALUES ( ' I am live ! ' ,

field ) VALUES ( ':) ', ' May cool
HadoopDB:
Listing 5.44:
$ java - cp $CLASSPATH : hadoopdb . jar

edu . yale . cs . hadoopdb . benchmark . GrepTaskDB - pattern % May %
- output padraig - hadoopdb . config . file
/ opt / hadoop / conf / HadoopDB . xml
96
5.3.
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
HadoopDB
padraig
01.11.2010 23:14:45 edu . yale . cs . hadoopdb . exec . DBJobBase
initConf
INFO : SELECT key1 , field FROM grep WHERE field LIKE '%% May %% ';
01.11.2010 23:14:46 org . apache . hadoop . metrics . jvm . JvmMetrics
init
INFO : Initializing JVM Metrics with processName = JobTracker ,
sessionId =
configureCommandLineOptions
WARNING : Use GenericOptionsParser for parsing the arguments .
Applications should implement Tool for the same .
monitorAndPrintJob
INFO : Running job : job_local_0001
01.11.2010 23:14:46
getConnection
INFO : Data locality failed for leo - pgsql
01.11.2010 23:14:46
getConnection
INFO : Task from leo - pgsql is connecting to chunk 0 on host
localhost with db url jdbc : postgresql :// localhost :5434/ grep0
01.11.2010 23:14:47 org . apache . hadoop . mapred . MapTask
runOldMapper
INFO : numReduceTasks : 0
01.11.2010 23:14:47
INFO : DB times ( ms ) : connection = 181 , query execution = 22 ,
row retrieval = 96
01.11.2010 23:14:47
INFO : Rows retrieved = 4
01.11.2010 23:14:47 org . apache . hadoop . mapred . Task done
INFO : Task : attempt_local_0001_m_000000_0 is done . And is in
the process of commiting
01.11.2010 23:14:47
INFO :
01.11.2010 23:14:47 org . apache . hadoop . mapred . Task commit
INFO : Task attempt_local_0001_m_000000_0 is allowed to commit
now
01.11.2010 23:14:47
org . apache . hadoop . mapred . FileOutputCommitter commitTask
INFO : Saved output of task ' attempt_local_0001_m_000000_0 ' to
file :/ home / hadoop / padraig
01.11.2010 23:14:47
INFO :
01.11.2010 23:14:47 org . apache . hadoop . mapred . Task sendDone
INFO : Task ' attempt_local_0001_m_000000_0 ' done .
97
5.3.
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
HadoopDB

monitorAndPrintJob
INFO : map 100% reduce 0%
monitorAndPrintJob
INFO : Job complete : job_local_0001
INFO : Counters : 6
INFO :
FileSystemCounters
INFO :
FILE_BYTES_READ =141345
INFO :
FILE_BYTES_WRITTEN =153291
INFO :
Map - Reduce Framework
INFO :
Map input records =4
INFO :
Spilled Records =0
INFO :
Map input bytes =4
INFO :
Map output records =4
01.11.2010 23:14:47 edu . yale . cs . hadoopdb . exec . DBJobBase run
INFO :
JOB TIME : 2332 ms .
May. . :
Listing 5.45:
1
2
3
4
5
6
$ cd padraig
$ cat part -00000
I am live ! Maybe
I am live ! Maybewqe
I am live ! Maybewqesad
:) May cool string !
PostgreSQL . HadoopDB
PostgreSQL,
PostgreSQL, shared-nothing .
HadoopDB
http://hadoopdb.sourceforge.net/guide/quick_start_guide.html.
, Hive,
HadoopDB. ,
98
5.4.
c Hadoop.
Hadoop HaddopDB.
HadoopDB Hadoop. ,
.
HadoopDB , , Hadoop. HadoopDB,
, , , PostgreSQL
, , PostgreSQL . , Hadoop Hive
.
HadoopDB
Hadoop , , , ,
MapReduce. HadoopDB Hadoop
( ) HadoopDB
.
5.4
.
PostgreSQL , , . ,
, .
99
6
PgPool-II
.

6.1
pgpool-II , PostgreSQL
PostgreSQL. :

pgpool-II PostgreSQL
(.. , , ).
.
pgpool-II PostgreSQL. 2 , .
, SELECT
. pgpool-II
PostgreSQL SELECT
,
.
PostgreSQL. .
100
6.2.

PostgreSQL,
. ,
, . pgpool-II
, .

,
.
.
pgpool-II PostgreSQL
. , () pgpool-II PostgreSQL, () pgpool-II . pgpool-II
, , ,
, pgpool-II .
http://pgpool.projects.postgresql.org/pgpoolII/doc/tutorial-en.html.
6.2 !

pgpool-II
.
pgpool-II
pgpool-II . , , .
Listing 6.1: pgpool-II
1
2
3
./ configure
make
make install
congure
.
101
6.2.
congure -, , , . pgpool-II - /usr/local.

make , make install
. .
: pgpool-II libpq
PostgreSQL 7.4 (3 ).
congure ,
libpq 3 .
configure : error : libpq is not installed or libpq is old

3 , - , libpq, , congure.
congure libpq /usr/local/pgsql.
PostgreSQL /usr/local/pgsql
with-pgsql with-pgsqlincludedir with-pgsql-libdir congure.
Linux pgpool-II
. Ubuntu Linux, , :
sudo aptitude install pgpool2

pgpool-II pgpool.conf. : = .
pgpool-II pgpool.conf.sample. pgpool.conf, .
Listing 6.4:
cp / usr / local / etc / pgpool . conf . sample / usr / local / etc / pgpool . conf
pgpool-II localhost 9999.
, listen_addresses *.
Listing 6.5:
1
2
listen_addresses = ' localhost '

port = 9999
102
6.2.
- .
Ubuntu Linux /etc/pgpool.conf.
PCP
pgpool-II
, pgpool-II .. . PCP, . PostgreSQL.
pcp.conf. , (:). .
md5.
Listing 6.6: PCP
postgres : e8a48653851e28c69d0506508fb27fc5
pgpool-II pcp.conf.sample.
pcp.conf .
Listing 6.7: PCP
$ cp / usr / local / etc / pcp . conf . sample / usr / local / etc / pcp . conf
Ubuntu Linux /etc/pcp.conf.
md5 pg_md5,
pgpool-II. pg_md5
md5 .
, postgres
pg_md5 md5 .
Listing 6.8: PCP
1
2
$ / usr / bin / pg_md5 postgres

e8a48653851e28c69d0506508fb27fc5
PCP , pgpool.conf pcp_port.
- pcp_port
9898 .
Listing 6.9: PCP
pcp_port = 9898
103
6.2.

PostgreSQL pgpoolII. pgpool-II
. ,
.
,
pgpool-II.

pgpool-II 5432, 5433, 5434 . pgpool-II pgpool.conf
.
Listing 6.10:
1
2
3
4
5
6
7
8
9
backend_hostname0 = ' localhost '

backend_port0 = 5432
backend_weight0 = 1
backend_weight1 = 1
backend_weight2 = 1
backend_hostname, backend_port, backend_weight , .
0 (.. 0, 1, 2).
backend_weight 1,
SELECT .
/ pgpool-II
pgpool-II .
Listing 6.11:
pgpool
, ,
pgpool .
pgpool, -n
pgpool. pgpool-II -
.
Listing 6.12:
pgpool -n &
104
6.3.
, , , .
Listing 6.13:
pgpool -n -d > / tmp / pgpool . log 2 >&1 &

-d .

/tmp/pgpool.log. , ,
. , , cronolog.
Listing 6.14:
1
2
3
pgpool -n 2 >&1 | / usr / sbin / cronolog

-- hardlink =/ var / log / pgsql / pgpool . log
'/ var / log / pgsql /% Y -% m -% d - pgpool . log ' &
pgpool-II, .
Listing 6.15:
pgpool stop
- , pgpool-II .
pgpool-II , .
Listing 6.16:
pgpool -m fast stop
6.3
.
, 6.2. !,
.
pgbench.

true replication_mode pgpool.conf.
105
6.3.
Listing 6.17:
replication_mode = true
replication_mode true, pgpool-II
.
load_balance_mode true, pgpool-II SELECT .
Listing 6.18:
load_balance_mode = true
replication_mode load_balance_mode.

, pgpool.conf, pgpoolII . / pgpool-II.
pgpool.conf pgpool-II,
.
, . bench_replication.
. createdb pgpool-II
.
Listing 6.19:
createdb -p 9999 bench_replication

pgbench -i. -i .
Listing 6.20:
pgbench -i -p 9999 bench_replication

, pgbench -i.
,
.

branches
tellers
10
accounts
100000
history

shell.
106
6.4.
branches, tellers, accounts history

(5432, 5433, 5434).
Listing 6.21:
1
2
3
4
5
6
7
8
for port in 5432 5433 5434; do

>
echo $port
>
for table_name in branches tellers accounts history ; do
>
echo $table_name
>
psql -c " SELECT count (*) FROM $table_name " -p \
>
$port bench_replication
>
done
> done
6.4
. ( partitioning . ). ,
.
pgpool-II ,
(System Database) ( SystemDB).
SystemDB , . SystemDB
dblink.
,
6.2. !,
.
pgbench.

parallel_mode true pgpool.conf.
Listing 6.22:
parallel_mode = true
parallel_mode true . pgpool-II SystemDB
.
107
6.4.
SystemDB dblink pgpoolII. , listen_addresses

pgpool-II .
Listing 6.23:
listen_addresses = '* '

: , , ,
. , - ,
bench_replication, 6.3.
.
Listing 6.24:
1
2
replication_mode = true
load_balance_mode = false
Listing 6.25:
1
2
replication_mode = false
load_balance_mode = true
parallel_mode load_balance_mode
true, listen_addresses *, replication_mode false.
SystemDB
, .
, dblink , .
dist_def . ,
, pgpool-II .
SystemDB 5432.
SystemDB
Listing 6.26: SystemDB
1
2
3
4
5
6
system_db_hostname = ' localhost '

system_db_port = 5432
system_db_dbname = ' pgpool '
system_db_schema = ' pgpool_catalog '
system_db_user = ' pgpool '
system_db_password = ''
108
6.4.
,
- pgpool.conf. pgpool pgpool
pgpool.
Listing 6.27: SystemDB
1
2
createuser -p 5432 pgpool

createdb -p 5432 -O pgpool pgpool
dblink
dblink pgpool. dblink
contrib PostgreSQL.
dblink .
Listing 6.28: dblink
1
2
USE_PGXS =1 make -C contrib / dblink

USE_PGXS =1 make -C contrib / dblink install
dblink
dblink pgpool. PostgreSQL
/usr/local/pgsql, dblink.sql ( ) /usr/local/pgsql/share/contrib. dblink.
Listing 6.29: dblink
psql -f / usr / local / pgsql / share / contrib / dblink . sql -p 5432

pgpool
dist_def
dist_def, . pgpool-II
, system_db.sql
/usr/local/share/system_db.sql ( - /usr/local).
system_db.sql , dist_def.
dist_def.
Listing 6.30: dist_def
$ psql -f / usr / local / share / system_db . sql -p 5432 -U pgpool

pgpool
109
6.4.
system_db.sql, dist_def,
pgpool_catalog. system_db_schema
, , system_db.sql.
dist_def .
.
Listing 6.31: dist_def
1
2
3
4
5
6
7
8
9
10
11
12
CREATE TABLE pgpool_catalog . dist_def (

dbname text , --
schema_name text , --
table_name text , --
col_name text NOT NULL CHECK ( col_name = ANY ( col_list ) ) ,
--
col_list text [] NOT NULL , --
type_list text [] NOT NULL , --
dist_def_func text NOT NULL ,
--
PRIMARY KEY ( dbname , schema_name , table_name )
);
, dist_def, .
(col_name, dist_def_func)
- (dbname, schema_name, table_name,
col_list, type_list)

.
col_name. dist_def_func
, col_name
,
, .
- .

, -,
.
replicate_def
,
SQL, dist_def
, , ,
replicate_def. replicate_def system_db.sql dist_def.
replicate_def .
110
6.4.
Listing 6.32: replicate_def
1
2
3
4
5
6
7
8
CREATE TABLE pgpool_catalog . replicate_def (

dbname text , --
schema_name text , --
table_name text , --
col_list text [] NOT NULL , --
type_list text [] NOT NULL , --
PRIMARY KEY ( dbname , schema_name , table_name )
);

, pgbench, . pgbench -i -s 3 (.. 3).
bench_parallel.
sample pgpool-II
dist_def_pgbench.sql.
pgbench. pgpool-II.
Listing 6.33:
psql -f sample / dist_def_pgbench . sql -p 5432 pgpool

dist_def_pgbench.sql.
dist_def_pgbench.sql
dist_def. accounts.
- aid.
Listing 6.34:
1
2
3
4
5
6
7
8
9
10
INSERT INTO pgpool_catalog . dist_def VALUES (

' bench_parallel ' ,
' public ' ,
' accounts ' ,
' aid ' ,
ARRAY [ ' aid ' , ' bid ' , ' abalance ' , ' filler '] ,
ARRAY [ ' integer ' , ' integer ' , ' integer ' ,
' character (84) '] ,
' pgpool_catalog . dist_def_accounts '
);
accounts. , . SQL (, PL/pgSQL, PL/Tcl, ..).
111
6.4.
accounts
3, aid 1 300000.

.
SQL- .
Listing 6.35:
1
2
3
4
5
6
7
8

pgpool_catalog . dist_def_branches ( anyelement )
RETURNS integer AS $$
SELECT CASE WHEN $1 > 0 AND $1 <= 1 THEN 0
WHEN $1 > 1 AND $1 <= 2 THEN 1
ELSE 2
END ;
$$ LANGUAGE sql ;

.
pgbench branches tellers.
, accounts , branches tellers.
Listing 6.36:
1
2
3
4
5
6
7
9
10
11
12
13
14
15
INSERT INTO pgpool_catalog . replicate_def VALUES (

' public ' ,
' branches ' ,
ARRAY [ ' bid ' , ' bbalance ' , ' filler '] ,
ARRAY [ ' integer ' , ' integer ' , ' character (88) ']
);
INSERT INTO pgpool_catalog . replicate_def VALUES (
' public ' ,
' tellers ' ,
ARRAY [ ' tid ' , ' bid ' , ' tbalance ' , ' filler '] ,
ARRAY [ ' integer ' , ' integer ' , ' integer ' , ' character (84) ']
);
Replicate_def_pgbench.sql sample. psql ,
, , .
Listing 6.37:
psql -f sample / replicate_def_pgbench . sql -p 5432 pgpool

112
6.4.

, pgpool.conf, pgpoolII . / pgpool-II.
pgpool.conf pgpool-II,
.
, .
bench_parallel.
. createdb pgpool-II
.
Listing 6.38:
createdb -p 9999 bench_parallel

pgbench -i -s 3. -i .
-s .
Listing 6.39:
pgbench -i -s 3 -p 9999 bench_parallel

.
SELECT pgpool-II
. bench_parallel .

branches
tellers
30
accounts
300000
history
pgpool-II shell.
accounts
5432, 5433, 5434 9999.
Listing 6.40:
1
2
3
4
5
for port in 5432 5433 5434 i 9999; do

>
echo $port
>
psql -c " SELECT min ( aid ) , max ( aid ) FROM accounts " \
>
-p $port bench_parallel
> done
113
6.5.
Master-slave
6.5 Master-slave
pgpool-II ( Slony-I, Londiste).
. master_slave_mode load_balance_mode true. pgpool-II INSERT/UPDATE/DELETE
Master DB (1 ), SELECT , .
, DDL DML . SELECT ,
/*NO LOAD BALANCE*/
SELECT.
Master/Slave replication_mode false,
master_slave_mode true.
master-slave ,
, pgpoolII. PostgreSQL, pgpool-II ( ),
( ).
, , ( recovery.conf, trigger_le),
PostgreSQL .
:
Listing 6.41: PostgreSQL
1
2
3
4
5
6
7
8
9
11
12
13
15
16
17
# ! / bin / sh
# Failover command for streming replication .
# This script assumes that DB node 0 is primary , and 1 is
standby .
#
# If standby goes down , does nothing . If primary goes down ,
create a
# trigger file so that standby take over primary node .
#
# Arguments : $1 : failed node id . $2 : new master hostname . $3 :
path to
# trigger file .
failed_node = $1
new_master = $2
trigger_file = $3
# Do nothing if standby goes down .
if [ $failed_node = 1 ]; then
exit 0;
114
6.6.
18
20
21
23
fi
# Create trigger file .
/ usr / bin / ssh -T $new_master / bin / touch $trigger_file
exit 0;
: ,
.
failover_stream.sh pgpool.conf :
Listing 6.42:
failover_command = '/ path_to_script / failover_stream . sh % d % H

/ tmp / trigger_file '
/tmp/trigger_le , recovery.conf.
, ,
.
6.6
pgpool-II, ,
pgpool. . ,
.
. pgpool, .
:
CHECKPOINT;
;
, ;
CHECKPOINT;
;
postmaster ( pgpool_remote_start);
;
backend_data_directory
PostgreSQL .
recovery_user
PostgreSQL.
115
6.6.
recovery_password
PostgreSQL.
recovery_1st_stage_command
.
- . ,
recovery_1st_stage_command = 'some_script', pgpool-II
$PGDATA/some_script. , pgpool-II recovery_1st_stage.
recovery_2nd_stage_command
.
- . ,
recovery_2st_stage_command = 'some_script', pgpool-II
$PGDATA/some_script. , pgpool-II recovery_2st_stage.
, pgpool-II ,
.
master-slave , PostgreSQL.
, . PostgreSQL
pgpool-II ( ).
:
recovery_user. postgres.
Listing 6.43: recovery_user
recovery_user = ' postgres '

recovery_password recovery_user .
Listing 6.44: recovery_password
recovery_password = ' some_password '

recovery_1st_stage_command.
basebackup.sh ($PGDATA),
. :
116
6.6.
Listing 6.45: basebackup.sh
1
2
3
4
5
6
7
9
11
12
13
15
17
# ! / bin / sh
# Recovery script for streaming replication .
# This script assumes that DB node 0 is primary , and 1 is
standby .
#
datadir = $1
desthost = $2
destdir = $3
psql -c " SELECT pg_start_backup ( ' Streaming Replication ' ,
true ) " postgres
rsync -C -a -- delete -e ssh -- exclude postgresql . conf
-- exclude postmaster . pid \
-- exclude postmaster . opts -- exclude pg_log -- exclude
pg_xlog \
-- exclude recovery . conf $datadir / $desthost : $destdir /
ssh -T localhost mv $destdir / recovery . done
$destdir / recovery . conf
psql -c " SELECT pg_stop_backup () " postgres
,
rsync .
SSH , recovery_user
.
:
Listing 6.46: recovery_1st_stage_command
recovery_1st_stage_command = ' basebackup . sh '

recovery_2nd_stage_command . ,
, basebackup.sh,
WAL .
C SQL
.
Listing 6.47: C SQL
1
2
3
4
$
$
$
$
cd pgpool - II - x . x . x / sql / pgpool - recovery

make
make install
psql -f pgpool - recovery . sql template1
117
6.7.
. pcp_recovery_node
.
6.7
PgPool-II , PostgreSQL.
118
7

,
. .
?

7.1
( ) , ,
PostgreSQL. Windows,
.
-, .
, :
PgBouncer
Pgpool
7.2 PgBouncer
PostgreSQL Skype.
.
Session Pooling.
;
.
Transaction Pooling. . PgBouncer ,

, .
119
7.3.
Statement Pooling. . .
, .
(prepared statements) .
PgBouncer :
( 2 );
;
.
:
Listing 7.1: PgBouncer
pgbouncer [ - d ][ - R ][ - v ][ - u user ] < pgbouncer . ini >

:
1
2
3
4
5
6
7
8
9
10
[ databases ]
template1 = host =127.0.0.1 port =5432 dbname = template1
[ pgbouncer ]
listen_port = 6543
listen_addr = 127.0.0.1
auth_type = md5
auth_file = userlist . txt
logfile = pgbouncer . log
pidfile = pgbouncer . pid
admin_users = someuser
userlist.txt :someuser same_password_as_in_server
pgbouncer:
psql -h 127.0.0.1 -p 6543 pgbouncer

SHOW.
7.3 PgPool-II vs PgBouncer

. PgBouncer , PgPool-II. , PgPool-II ( ),
PgBouncer.
120
7.3.
PgBouncer , PgPool-II
PgBouncer
PgBouncer (
)
PgBouncer PgPool-II .
121
8
PostgreSQL
- , - .

8.1
, , . SELECT PostgreSQL.
, , , .
SQL ,
, , . PostgreSQL
. ? -,
. ? (MVCC MultiVersion Concurrency
Control) , , ,
. , ,
. , , . -, , ,
, .
( ),

.
122
8.2.
Pgmemcache
, , . PostgreSQL:
Pgmemcache ( memcached)
Pgpool-II (query cache)
8.2 Pgmemcache
1
Memcached
, -
-.
.
-.
.
,
memcached .
Pgmemcache PostgreSQL API libmemcached

memcached. PostgreSQL
, , memcached.
, .
2.0.4 pgmemcache .
Pgmemcache PostgreSQL
8.4 ( 9.0 ), Ubuntu Server
10.10. Pgmemcache , PostgreSQL
PGXS ( , Linux PGXS). memcached libmemcached
0.38.
,
Pgmemcache:

:
Listing 8.1:
http://memcached.org/
http://pgfoundry.org/projects/pgmemcache/
3 http://pgfoundry.org/frs/download.php/2672/pgmemcache_2.0.4.tar.bz2
1
2
123
8.2.
1
2
Pgmemcache
$ make
$ sudo make install
deb ( Debian, Ubuntu)

, Debian Ubuntu,
deb
PostgreSQL:
Listing 8.2: deb
1
2
3
4
$ sudo apt - get install libmemcached - dev

postgresql - server - dev -8.4 libpq - dev devscripts yada
flex bison
$ make deb84
# deb
$ sudo dpkg -i ../ postgresql - pgmemcache -8.4*. deb
2.0.4 yada deb
:
Listing 8.3: deb
1
2
Cannot recognize source name in ' debian / changelog ' at

/ usr / bin / yada line 145 , < CHANGELOG > line 1.
make : *** [ deb84 ] 9

debian/changelog , :
Listing 8.4: deb
1
2
4
$PostgreSQL : pgmemcache / debian / changelog , v 1.2 2010/05/05

19:56:50 ormod Exp $ <----
pgmemcache (2.0.4) unstable ; urgency = low
* v2 .0.4
, deb .
Pgmemcache ( Pgmemcache)
:
Listing 8.5:
124
8.2.
1
2
3
4
5
Pgmemcache
% psql [ mydbname ] [ pguser ]

[ mydbname ]= # BEGIN ;
[ mydbname ]= # \ i
/ usr / local / postgresql / share / contrib / pgmemcache . sql
# Debian : \ i
/ usr / share / postgresql /8.4/ contrib / pgmemcache . sql
[ mydbname ]= # COMMIT ;
memcached memcache_server_add
. . memcached PostgreSQL.
, postgresql.conf :
pgmemcache shared_preload_libraries ( pgmemcache PostgreSQL)
pgmemcache custom_variable_classes (
pgmemcache)
pgmemcache.default_servers, host:port
(port - ) . :
Listing 8.6: default_servers
pgmemcache . default_servers = '127.0.0.1 ,

192.168.0.20:11211 ' # memcached
pgmemcache
pgmemcache.default_behavior.
libmemcached. :
Listing 8.7: pgmemcache
pgmemcache . default_behavior = ' BINARY_PROTOCOL :1 '
PostgreSQL
memcached.
pgmemcache, memcached :
. memcached :
Listing 8.8: memcache_stats
1
2
3
pgmemcache = # SELECT memcache_stats () ;

memcache_stats
--------------------------125
8.2.
Pgmemcache
8.1: pgmemcache
memcache_server_add('hostname:port'::TEXT) memcached memcache_server_add('hostname'::TEXT)

. , 11211.
memcache_add(key::TEXT, value::TEXT,
, expire::TIMESTAMPTZ)
.
memcache_add(key::TEXT, value::TEXT,
expire::INTERVAL)
memcache_add(key::TEXT, value::TEXT)
newval = memcache_decr(key::TEXT,
decrement::INT4)
,
newval = memcache_decr(key::TEXT)
( ).
.
memcache_delete(key::TEXT,
. hold_timer::INTERVAL)
,
memcache_delete(key::TEXT)
.
memcache_ush_all()
value = memcache_get(key::TEXT)
memcache_get_multi(keys::TEXT[])
memcache_get_multi(keys::BYTEA[])
newval = memcache_incr(key::TEXT,
increment::INT4)
newval = memcache_incr(key::TEXT)
memcache_replace(key::TEXT, value::TEXT,
expire::TIMESTAMPTZ)
memcache_replace(key::TEXT, value::TEXT,
expire::INTERVAL)
memcache_replace(key::TEXT, value::TEXT)
5
6
7
8
9
10
11
memcached .
. NULL,
,
.
.
=.
,
( ).
.
.
memcache_set(key::TEXT, value::TEXT,
expire::TIMESTAMPTZ)
memcache_set(key::TEXT, value::TEXT,
expire::INTERVAL)
memcache_set(key::TEXT, value::TEXT)
.
.
stats = memcache_stats()

memcached.
Server : 127.0.0.1 (11211)

pid : 1116
uptime : 70
time : 1289598098
version : 1.4.5
pointer_size : 32
rusage_user : 0.0
126
8.2.
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
29
Pgmemcache
rusage_system : 0.24001
curr_items : 0
total_items : 0
bytes : 0
curr_connections : 5
total_connections : 7
connection_structures : 6
cmd_get : 0
cmd_set : 0
get_hits : 0
get_misses : 0
evictions : 0
bytes_read : 20
bytes_written : 782
limit_maxbytes : 67108864
threads : 4
(1 row )
memcached :
Listing 8.9:
1
2
3
4
5
7
8
9
10
11
pgmemcache = # SELECT memcache_add ( ' some_key ' , ' test_value ') ;

memcache_add
-------------t
(1 row )
pgmemcache = # SELECT memcache_get ( ' some_key ') ;
memcache_get
-------------test_value
(1 row )
memcached ( ):
Listing 8.10:
1
2
3
4
5
7
8
9
10
11
13
14
pgmemcache = # SELECT memcache_add ( ' some_seq ' , '10 ') ;

memcache_add
-------------t
(1 row )
pgmemcache = # SELECT memcache_incr ( ' some_seq ') ;
memcache_incr
--------------11
(1 row )
pgmemcache = # SELECT memcache_incr ( ' some_seq ') ;
memcache_incr
127
8.2.
15
16
17
19
20
21
22
23
25
26
27
28
29
31
32
33
34
35
37
38
39
40
41
Pgmemcache
--------------12
(1 row )
pgmemcache = # SELECT memcache_incr ( ' some_seq ' , 10) ;
memcache_incr
--------------22
(1 row )
pgmemcache = # SELECT memcache_decr ( ' some_seq ') ;
memcache_decr
--------------21
(1 row )
pgmemcache = # SELECT memcache_decr ( ' some_seq ') ;
memcache_decr
--------------20
(1 row )
pgmemcache = # SELECT memcache_decr ( ' some_seq ' , 6) ;
memcache_decr
--------------14
(1 row )
pgmemcache , ,
.
, memcached ( ),
, .
:
Listing 8.11:
1
2
3
4
5
6
7
8
CREATE OR REPLACE FUNCTION auth_passwd_upd () RETURNS TRIGGER

AS $$
BEGIN
IF OLD . passwd != NEW . passwd THEN
PERFORM memcache_set ( ' user_id_ ' || NEW . user_id ||
' _password ' , NEW . passwd ) ;
END IF ;
RETURN NEW ;
END ;
$$ LANGUAGE ' plpgsql ';
:
Listing 8.12:
CREATE TRIGGER auth_passwd_upd_trg AFTER UPDATE ON passwd FOR

EACH ROW EXECUTE PROCEDURE auth_passwd_upd () ;
128
8.2.
Pgmemcache
(!!!) .
:
Listing 8.13:
1
2
3
4
5
6
7
CREATE OR REPLACE FUNCTION auth_passwd_upd () RETURNS TRIGGER

AS $$
BEGIN
IF OLD . passwd != NEW . passwd THEN
PERFORM memcache_delete ( ' user_id_ ' || NEW . user_id ||
' _password ') ;
END IF ;
RETURN NEW ;
END ; $$ LANGUAGE ' plpgsql ';
:
Listing 8.14:
CREATE TRIGGER auth_passwd_del_trg AFTER DELETE ON passwd FOR

EACH ROW EXECUTE PROCEDURE auth_passwd_upd () ;
, memcached ( ) .
PostgreSQL Pgmemcache
memcached ,
. memcached,
SQL , PostgreSQL.
129
9.1
PostgreSQL
.
.
9.2 PostGIS
: Open Source
: http://www.postgis.org/
PostGIS PostgreSQL.
PostGIS PostgreSQL
(),
, ESRI SDE Oracle. PostGIS
OpenGIS " SQL"
.
9.3 PostPic
: Open Source
: http://github.com/drotiro/postpic
PostPic PostgreSQL, , PostGIS . image,
(, , ..) (, , ).
130
9.4.
OpenFTS
9.4 OpenFTS
: Open Source
: http://openfts.sourceforge.net/
OpenFTS (Open Source Full Text Search engine)
PostgreSQL , . , .
9.5 PL/Proxy
: Open Source
: http://pgfoundry.org/projects/plproxy/
PL/Proxy - . 5.2 .
9.6 Texcaller
: Open Source
: http://www.profv.de/texcaller/
Texcaller TeX, . C,
, , TeX.
TeX NULL,
. , , NOTICEs.
9.7 Pgmemcache
: Open Source
: http://pgfoundry.org/projects/pgmemcache/
Pgmemcache PostgreSQL API libmemcached
memcached. PostgreSQL
, , memcached.
8.2 .
9.8 Prex
: Open Source
: http://pgfoundry.org/projects/prex
131
9.9.
pgSphere
Prex (prex @> text).

LIKE prex%? GiST (, ).
9.9 pgSphere
: Open Source
: http://pgsphere.projects.postgresql.org/
pgSphere PostgreSQL ,
.
( PostGIS) .
9.10
PostgreSQL . PostgreSQL , , ,
.
132
10
PostgreSQL
,
, ,

- ,
.

10.1
.
, ,
, - .
PostgreSQL . !
, ,
, PostgreSQL .
, , (DELETE, DROP),
( ).
PostgreSQL:
SQL ;
;
;
10.2 SQL
SQL.

133
10.2.
SQL
, . PostgreSQL
pg_dump.
pg_dump:
Listing 10.1: pg_dump
pg_dump dbname > outfile

:
Listing 10.2:
psql dbname < infile

dbname . ,
, ( , ). ,
, :
Listing 10.3:
psql -- set ON_ERROR_STOP = on dbname < infile

,
:
Listing 10.4:
pg_dump -h host1 dbname | psql -h host2 dbname

ANALYZE, .
, , ,
? PostgreSQL pg_dumpall. pg_dumpall
PostgreSQL:
pg_dumpall > outfile

:
psql -f infile postgres
134
10.2.
SQL
SQL
, pg_dump. , pg_dump
. Unix, . :
.
, GZIP:
pg_dump dbname | gzip > filename . gz

:
gunzip -c filename . gz | psql dbname
cat filename . gz | gunzip | psql dbname
split.
split , . , 1 :
pg_dump dbname | split -b 1 m - filename

:
cat filename * | psql dbname
pg_dump
PostgreSQL Zlib,
.
GZIP, :
135
10.3.
pg_dump - Fc dbname > filename

psql ,
pg_restore:
pg_restore -d dbname filename
, split
.
10.3
, PostgreSQL
. :
tar - cf backup . tar / usr / local / pgsql / data

, , , , SQL :
PostgreSQL , ,
(PostgreSQL , ). ,
PostgreSQL.
, (snapshot)
( PostgreSQL). PostgreSQL
. , , , ,
. PostgreSQL , ,
WAL. ,
( WAL ). ,
PostgreSQL ,
136
10.4.
(!!!). ,
.
rsync.
rsync PostgreSQL (PostgreSQL
). PostgreSQL
rsync. rsync , ,
, .
.
10.4
PostgreSQL (Write Ahead
Log, WAL) pg_xlog , . .
PostgreSQL: ,
. ,
:
WAL . ,
, WAL .
, , :
.

( , ).
WAL
, ,
PostgreSQL
( ).
, . ,
WAL .
137
10.5.
. WAL pg_xlog.
postgresql.conf:
Listing 10.15:
1
2
3
archive_mode = on # enable archiving

archive_command = ' cp -v % p / data / pgsql / archives /% f '
archive_timeout = 300 # timeout to close buffers
( )
. rsync.
, ,
.
Listing 10.16: WAL
1
2
rsync - avz -- delete prod1 :/ data / pgsql / archives / \

/ data / pgsql / archives / > / dev / null
, pg_xlog
PostgreSQL ( ). PostgreSQL recovery.conf
:
Listing 10.17: recovery.conf
restore_command = ' cp / data / pgsql / archives /% f " % p " '

PostgreSQL , (,
,
).
http://www.postgresql.org/docs/9.0/static/continuous-archiving.html.
10.5
, , , . ,
PostgreSQL (, ).
138
11

PostgreSQL
, (),
.
-
, , .

11.1
,

. -
( , ).
, SQL , , PostgreSQL ,
. , . :
?
( ..),
, , .
, ,
.

, - - , . ,
:
139
11.2.
;
;
,
, ( Twitter Facebook
). , .
11.2
, , .
, .. ,
,
.

PgPool-II v.3 + Postgresql v.9 Streaming Replication ,
. :

(failover)

.
pgpool-II

PgPool-II v.3 + Postgresql Slony
, Slony. :

1
2
(failover)

.
pgpool-II
Postgresql 9
http://pgpool.projects.postgresql.org/contrib_docs/simple_sr_setting/index.html
http://www.slideshare.net/leopard_me/postgresql-5845499
140
11.3.
11.3
,
( Google Analytics).
( ).

. PgQ.
PgQ , PostgreSQL. Skype. Londiste ( 4.3).
:
PostgreSQL
,
PgQ , ,

(batches)
API SQL

RabbitMQ. RabbitMQ
, (Message Oriented Middleware) AMQP (Advanced Message Queuing Protocol). RabbitMQ
Mozilla Public License. RabbitMQ Open
Telecom Platform, Erlang.
141
12

(Performance Snippets)

.
. -
.
, ,
.
(House M.D.), 1 1
12.1
PostgreSQL,
, ( ).
, PostgreSQL.
, PostgreSQL ,
.
,

https://github.com/le0pard/postgresql_book/issues
142
12.2.
12.2

( ). PostgreSQL >= 8.1.
Listing 12.1: . SQL
1
2
3
4
5
6
7
SELECT nspname || '. ' || relname AS " relation " ,

pg_size_pretty ( pg_relation_size ( C . oid ) ) AS " size "
FROM pg_class C
LEFT JOIN pg_namespace N ON ( N . oid = C . relnamespace )
WHERE nspname NOT IN ( ' pg_catalog ' , ' information_schema ')
ORDER BY pg_relation_size ( C . oid ) DESC
LIMIT 20;
: https://gist.github.com/910674
:
Listing 12.2: .
1
2
3
4
5
6
7
8
9
relation
|
size
- - - - - - - - - - - - - - - - - - - - - - - -+ - - - - - - - - - - - public . accounts
| 326 MB
public . accounts_pkey
| 44 MB
public . history
| 592 kB
public . tellers_pkey
| 16 kB
public . branches_pkey
| 16 kB
public . tellers
| 16 kB
public . branches
| 8192 bytes

. PostgreSQL >= 8.1.
Listing 12.3: . SQL
1
2
3
4
5
6
7
8
9
SELECT nspname || '. ' || relname AS " relation " ,

pg_size_pretty ( pg_total_relation_size ( C . oid ) ) AS
" total_size "
FROM pg_class C
LEFT JOIN pg_namespace N ON ( N . oid = C . relnamespace )
WHERE nspname NOT IN ( ' pg_catalog ' , ' information_schema ')
AND C . relkind <> 'i '
AND nspname !~ '^ pg_toast '
ORDER BY pg_total_relation_size ( C . oid ) DESC
LIMIT 20;
143
12.2.
:
Listing 12.4: .
1
2
3
4
5
6
7
8
relation
| total_size
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -+ - - - - - - - - - - - public . actions
| 4249 MB
public . product_history_records | 197 MB
public . product_updates
| 52 MB
public . import_products
| 34 MB
public . products
| 29 MB
public . visits
| 25 MB
count
. ,
count.
Listing 12.5: count. SQL
1
2
3
4
5
6
7
8
9
10
12
13
14
RETURN rows ;
END ;
$$ LANGUAGE plpgsql VOLATILE STRICT ;
17
Testing :
20
21
22
24
25
26
27
CREATE LANGUAGE plpgsql ;

CREATE FUNCTION count_estimate ( query text ) RETURNS integer AS
$$
DECLARE
rec
record ;
rows integer ;
BEGIN
FOR rec IN EXECUTE ' EXPLAIN ' || query LOOP
rows := substring ( rec . " QUERY PLAN " FROM '
rows =([[: digit :]]+) ') ;
EXIT WHEN rows IS NOT NULL ;
END LOOP ;
CREATE TABLE foo ( r double precision ) ;

INSERT INTO foo SELECT random () FROM generate_series (1 , 1000) ;
ANALYZE foo ;
# SELECT count (*) FROM foo WHERE r < 0.1;
count
------92
144
12.2.
28
30
31
32
33
34
(1 row )
# SELECT count_estimate ( ' SELECT * FROM foo WHERE r < 0.1 ') ;
count_estimate
---------------94
(1 row )
-
-
( DEFAULT).
Listing 12.6: - . SQL
1
2
3
4
5
6
7
8
9
CREATE OR REPLACE FUNCTION ret_def ( text , text , text ) RETURNS

text AS $$
SELECT
COLUMNS . column_default :: text
FROM
information_schema . COLUMNS
WHERE table_name = $2
AND table_schema = $1
AND column_name = $3
$$ LANGUAGE sql IMMUTABLE ;
:
Listing 12.7: - .
1
3
4
5
6
7
9
10
11
13
# SELECT ret_def ( ' schema ' , ' table ' , ' column ') ;
SELECT ret_def ( ' public ' , ' image_files ' , ' id ') ;
ret_def
----------------------------------------nextval ( ' image_files_id_seq ':: regclass )
(1 row )
SELECT ret_def ( ' public ' , ' schema_migrations ' , ' version ') ;
ret_def
--------(1 row )
145
12.2.

(random) ( ).
Listing 12.8: . SQL
1
2
3
4
5
CREATE OR REPLACE FUNCTION random ( numeric , numeric )

RETURNS numeric AS
$$
SELECT ( $1 + ( $2 - $1 ) * random () ) :: numeric ;
$$ LANGUAGE 'sql ' VOLATILE ;
:
Listing 12.9: .
1
2
3
4
5
7
8
9
10
11
SELECT random (1 ,10) :: int , random (1 ,10) ;

random |
random
- - - - - - - -+ - - - - - - - - - - - - - - - - - 6 | 5.11675184825435
(1 row )
SELECT random (1 ,10) :: int , random (1 ,10) ;
random |
random
- - - - - - - -+ - - - - - - - - - - - - - - - - - 7 | 1.37060070643201
(1 row )

, . , , , .
IBM 1960 .
,
.
http://en.wikipedia.org/wiki/Luhn_algorithm
SQL. ,
.
146
12.2.
Listing 12.10: . SQL
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
27
28
29
31
32
33
34
35
36
37
38
CREATE OR REPLACE FUNCTION luhn_verify ( int8 ) RETURNS BOOLEAN

AS $$
-- Take the sum of the
-- doubled digits and the even - numbered undoubled digits , and
see if
-- the sum is evenly divisible by zero .
SELECT
-- Doubled digits might in turn be two digits . In
that case ,
-- we must add each digit individually rather than
adding the
-- doubled digit value to the sum . Ie if the original
digit was
-- `6 ' the doubled result was `12 ' and we must add
`1+2 ' to the
-- sum rather than `12 '.
MOD ( SUM ( doubled_digit / INT8 '10 ' + doubled_digit %
INT8 '10 ') , 10) = 0
FROM
-- Double odd - numbered digits ( counting left with
-- least significant as zero ) . If the doubled digits end up
-- having values
-- > 10 ( ie they ' re two digits ) , add their digits together .
( SELECT
-- Extract digit `n ' counting left from least
significant
-- as zero
MOD ( ( $1 :: int8 / (10^ n ) :: int8 ) , 10:: int8 )
-- Double odd - numbered digits
* ( MOD (n ,2) + 1)
AS doubled_digit
FROM generate_series (0 , CEIL ( LOG ( $1 ) ) :: INTEGER - 1)
AS n
) AS doubled_digits ;
$$ LANGUAGE 'SQL '
IMMUTABLE
STRICT ;
COMMENT ON FUNCTION luhn_verify ( int8 ) IS ' Return true iff the
last digit of the
input is a correct check digit for the rest of the input
according to Luhn ' ' s
algorithm . ';
CREATE OR REPLACE FUNCTION luhn_generate_checkdigit ( int8 )
RETURNS int8 AS $$
SELECT
-- Add the digits , doubling even - numbered digits
( counting left
-- with least - significant as zero ) . Subtract the
remainder of
-- dividing the sum by 10 from 10 , and take the remainder
147
12.2.
39
40
41
42
43
44
45
46
47
48
49
50
52
53
54
56
57
58
59
60
61
62
64
65
66
67
68
69
70
71
73
74
75
-- of dividing that by 10 in turn .

(( INT8 '10 ' - SUM ( doubled_digit / INT8 '10 ' +
doubled_digit % INT8 '10 ') %
INT8 '10 ') % INT8 '10 ') :: INT8
FROM ( SELECT
-- Extract digit `n ' counting left from least
significant \
-- as zero
MOD ( ( $1 :: int8 / (10^ n ) :: int8 ) , 10:: int8 )
-- double even - numbered digits
* (2 - MOD (n ,2) )
AS doubled_digit
FROM generate_series (0 , CEIL ( LOG ( $1 ) ) :: INTEGER - 1)
AS n
) AS doubled_digits ;
$$ LANGUAGE 'SQL '
IMMUTABLE
STRICT ;
COMMENT ON FUNCTION luhn_generate_checkdigit ( int8 ) IS ' For the
input
value , generate a check digit according to Luhn ' ' s algorithm ';
CREATE OR REPLACE FUNCTION luhn_generate ( int8 ) RETURNS int8 AS
$$
SELECT 10 * $1 + luhn_generate_checkdigit ( $1 ) ;
$$ LANGUAGE 'SQL '
IMMUTABLE
STRICT ;
COMMENT ON FUNCTION luhn_generate ( int8 ) IS ' Append a check
digit generated
according to Luhn ' ' s algorithm to the input value . The input
value must be no
greater than ( maxbigint /10) . ';
CREATE OR REPLACE FUNCTION luhn_strip ( int8 ) RETURNS int8 AS $$
SELECT $1 / 10;
$$ LANGUAGE 'SQL '
IMMUTABLE
STRICT ;
COMMENT ON FUNCTION luhn_strip ( int8 ) IS ' Strip the least
significant digit from
the input value . Intended for use when stripping the check
digit from a number
including a Luhn ' ' s algorithm check digit . ';
:
Listing 12.11: .
1
2
Select luhn_verify (49927398716) ;

luhn_verify
148
12.2.
3
4
5
7
8
9
10
11
------------t
(1 row )
Select luhn_verify (49927398714) ;
luhn_verify
------------f
(1 row )

IN. , . :
: (2,6,4,10,25,7,9)
.. 2 2 2 6 6 4 4
Listing 12.12: . SQL
1
2
3
4
SELECT foo .* FROM foo

JOIN ( SELECT id . val , row_number () over () FROM
( VALUES (3) ,(2) ,(6) ,(1) ,(4) ) AS
id ( val ) ) AS id
ON ( foo . catalog_id = id . val ) ORDER BY row_number ;
VALUES(3),(2),(6),(1),(4)
foo ,
foo.catalog_id ( foo.catalog_id
IN(3,2,6,1,4))
,
, (. quine) (
), .
SQL PostgreSQL.
Listing 12.13:
1
2
select a || ' from ( select ' || quote_literal ( a ) || b || ', '

|| quote_literal ( b ) || ':: text as b ) as quine ' from
( select ' select a || '' from ( select '' || quote_literal ( a ) ||
b || '', '' || quote_literal ( b ) || ' ':: text as b ) as
149
12.2.
quine ' ' ':: text as a , ':: text as a ':: text as b ) as quine ;
LIKE
web2.0 .
LIKE some%, some ,
. ,
( ) .
LIKE 'bla%'
text_pattern_ops ( varchar_pattern_ops varchar). .
Listing 12.14: LIKE
1
2
3
4
5
6
7
8
9
10
11
12
14
15
16
17
18
19
20
21
22
23
27
28
29
30
31
prefix_test =# create table tags (

prefix_test (# tag
text primary key ,
prefix_test (# name
text not null ,
prefix_test (# shortname text ,
prefix_test (# status
char default 'S ' ,
prefix_test (#
prefix_test (# check ( status in ( 'S ' , 'R ') )
prefix_test (# ) ;
" tags_pkey " for table " tags "
CREATE TABLE
prefix_test =# CREATE INDEX i_tag ON tags USING
btree ( lower ( tag ) text_pattern_ops ) ;
CREATE INDEX
prefix_test =# create table invalid_tags (
prefix_test (# tag
text primary key ,
prefix_test (# name
text not null ,
prefix_test (# shortname text ,
prefix_test (# status
char default 'S ' ,
prefix_test (#
prefix_test (# check ( status in ( 'S ' , 'R ') )
prefix_test (# ) ;
" invalid_tags_pkey " for table " invalid_tags "
CREATE TABLE
prefix_test =# select count (*) from tags ;

count
------11966
(1 row )
150
12.2.
33
34
35
36
37
prefix_test =# select count (*) from invalid_tags ;

count
------11966
(1 row )
41
TEST STANDART LIKE
44
45
46
47
48
49
50
52
53
54
55
56
57
58
62
65
66
67
68
69
70
71
72
73
75
# EXPLAIN ANALYZE select * from invalid_tags where lower ( tag )

LIKE lower ( '0146% ') ;
QUERY PLAN
- -- -- -- -- -- - -- -- -- -- -- - -- -- -- -- - -- -- -- -- -- - -- -- -- -- -- - -- -- -- -- -- - -- -- -- -- -- - -Seq Scan on invalid_tags ( cost =0.00..265.49 rows =60
width =26) ( actual time =0.359..20.695 rows =1 loops =1)
Filter : ( lower ( tag ) ~~ '0146% ':: text )
Total runtime : 20.803 ms
(3 rows )
# EXPLAIN ANALYZE select * from invalid_tags where lower ( tag )
LIKE lower ( '0146% ') ;
QUERY PLAN
- -- -- -- -- -- - -- -- -- -- -- - -- -- -- -- - -- -- -- -- -- - -- -- -- -- -- - -- -- -- -- -- - -- -- -- -- -- - -Seq Scan on invalid_tags ( cost =0.00..265.49 rows =60
(3 rows )
TEST VARIANT WITH text_pattern_ops

# EXPLAIN ANALYZE select * from tags where lower ( tag ) LIKE
lower ( '0146% ') ;
QUERY
PLAN
- -- -- - -- -- - -- -- - -- -- -- - -- -- - -- -- - -- -- - -- -- -- - -- -- - -- -- - -- -- - -- -- - -- -- -- - -- -- - Bitmap Heap Scan on tags ( cost =5.49..97.75 rows =121
-> Bitmap Index Scan on i_tag ( cost =0.00..5.46 rows =120
Index Cond : (( lower ( tag ) ~ >=~ '0146 ':: text ) AND
( lower ( tag ) ~ <~ '0147 ':: text ) )
(5 rows )
# EXPLAIN ANALYZE select * from tags where lower ( tag ) LIKE
lower ( '0146% ') ;
151
12.2.
76
77
78
79
80
81
82
83
QUERY
PLAN
- -- -- - -- -- - -- -- - -- -- -- - -- -- - -- -- - -- -- - -- -- -- - -- -- - -- -- - -- -- - -- -- - -- -- -- - -- -- - Bitmap Heap Scan on tags ( cost =5.49..97.75 rows =121
-> Bitmap Index Scan on i_tag ( cost =0.00..5.46 rows =120
Index Cond : (( lower ( tag ) ~ >=~ '0146 ':: text ) AND
( lower ( tag ) ~ <~ '0147 ':: text ) )
(5 rows )
152

Postgresql v200b

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Postgresql v200b

Uploaded by

Copyright:

Available Formats

..

PostgreSQL: . (Sad Spirit)

Londiste , Sergey Konoplev gray.ru@gmail.com,

pgpool-II, Dmitry Stasyuk,

Up and Running with HadoopDB, Padraig O'Sullivan,

Streaming Replication ( ) . . . . . . . . . 114

10.2 SQL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133

7.4 ( IN/NOT IN).

8.0 , , CHECKPOINT VACUUM .

8.4 , , , EXISTS/NOT EXISTS .

9.0 , VACUUM/VACUUM FULL

Laptop, Celeron processor, 384 RAM, 25 : 12

Quad PIII server, 4 RAM, 40 , 150 , : 1

Quad Xeon server, 8 RAM, 200 , 300 ,

. , 14 32128 MB.

Free Space Map: VACUUM FULL

Free Space Map , .

shared_buers = 1/8 RAM ( 1/4);

pgtune -i $PGDATA / postgresql . conf \

-i, --input-config postgresql.conf,

. : DW, OLTP, Web,

EXPLAIN ANALYZE DELETE . . . 

PRIMARY KEY UNIQUE.

CREATE INDEX foo_name_first_idx

SELECT * FROM foo

CREATE INDEX scheta_neuplocheno ON scheta (id)

SELECT * FROM scheta WHERE NOT uplocheno AND ...;

SELECT count(*) FROM < >

9 RULE  PostgreSQL SQL, ,

Listing 2.2: SQL

SELECT count (*) FROM foo ;

SELECT reltuples FROM pg_class WHERE relname = ' foo ';

postgres =# select count (*) from ( select distinct i from g ) a ;

postgres =# select count (*) from ( select distinct i from g ) a ;

Listing 2.5: GROUP BY

postgres =# select count (*) from ( select i from g group by i ) a ;

log- PostgreSQL. pgFouine , . pgFouine PHP - ,

log_destination = ' syslog '

CHECK ( outletID BETWEEN 100 AND 200 )

CREATE TABLE my_logs (

CREATE TABLE my_logs2010m10

CREATE INDEX my_logs2010m10_logdate

CREATE OR REPLACE FUNCTION my_logs_insert_trigger ()

CREATE TRIGGER insert_my_logs_trigger

INSERT INTO my_logs ( user_id , logdate , data , some_state )

partitioning_test =# SELECT * FROM ONLY my_logs ;

partitioning_test =# SELECT * FROM my_logs ;

partitioning_test =# Select * from my_logs2010m10 ;

partitioning_test =# SELECT * FROM my_logs WHERE user_id = 2;

DROP TABLE my_logs2008m10 ;

DROP TABLE , DELETE. ,

ALTER TABLE my_logs2008m10 NO INHERIT my_logs ;

partitioning_test =# SET constraint_exclusion = off ;

Index Cond : ( logdate > ' 2010 -12 -01

partitioning_test =# SET constraint_exclusion = on ;

Slony-I1  Master-Slave , (cascading) (failover). Slony-I PostgreSQL INSERT/ DELETE/UPDATE

Mammoth Replicator7  Multi-Master .

Slony , PostgreSQL . Slony

Postgre INSERT/ DELETE/UPDATE

sudo aptitude install slony1 - bin

pgsql@customers_master$ createuser -a -d slony

anyuser@customers_slave$ psql -d customers \

pgsql@customers_slave$ createuser -a -d customers_owner

. , 14 32128 MB.

shared_buers = 1/8 RAM ( 1/4);

EXPLAIN ANALYZE DELETE . . .

9 RULE PostgreSQL SQL, ,

Slony-I1 Master-Slave , (cascading) (failover). Slony-I PostgreSQL INSERT/ DELETE/UPDATE

Mammoth Replicator7 Multi-Master .

wal_level = hot_standby WAL

archive_mode = on WAL archive_command .

master_db_datadir postgresql masterdb

Bucardo master-master master-slave PostgreSQL,

source source (master_db