PostgreSQL Master Slave Replication On Kubernetes

PostgreSQL Master Slave Replication on Kubernetes
Master-slave replication is a process in database management systems (DBMS) where data

from the master database is replicated to the slave databases which is widely used to
improve performance, scalability, and fault tolerance. Also a cost-effective solution for
managing data across distributed systems.
Today we will do hands-on project to deploy a postgreSQL master -slave replication using
microservice cluster system kubernetes. The main focus of this hands-on project is to deploy
PostgreSQL Master-slave replication deployment on kubernetes.
1
Author: Md Shafiqul Islam - [ Lead DevOps Architect ]

Linkedin: https://www.linkedin.com/in/uzzal2k5/
Email: uzzal2k5@gmail.com
WhatsApp: +8801715519132
DIRECTORY LISTING - YAML
Files / Folder Structure -
POSTGRES CONFIGMAP - postgres-configmap.yaml
Postgres Configmap will declare the DB name which has to be created on the spin of PODs.
YAML Configuration -
apiVersion: v1
kind: ConfigMap
metadata:
name: postgres-configmap
data:
POSTGRES_DB: trumpet-node-data
TIMESCALEDB_TELEMETRY: "off"
Apply Configuration -
# Create Credentials ConfigMap (cm)
kubectl apply -f postgres-configmap.yaml
kubectl get cm -o wide

WhatsApp: +8801715519132
POSTGRES SECRETS - postgres-secrets.yaml
Postgres Secrets will declare the secrets ( Credentials ) for the Database that will be use.
apiVersion: v1
kind: Secret
metadata:
name: postgres-secrets
type: Opaque
data:
POSTGRES_PASSWORD: cG9zdGdyZXM=
stringData:
POSTGRES_USER: postgres
# Create Credentials ConfigMap (cm)
kubectl apply -f postgres-secret.yaml
kubectl get secret -o wide
POSTGRES VOLUME - postgres-volume.yaml
Postgres Volume will be use to create all require volume and mapping the volume.
Postgres Master Data Persistent Volume Configmap -

# postgres master data configmap volume
apiVersion: v1
kind: PersistentVolume
metadata:
name: postgres-master-pv-configmap
spec:
capacity:
storage: 5Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
hostPath:
path: /mnt/master
nodeAffinity:
3

WhatsApp: +8801715519132
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- minikube # Node Name
Postgres Master Data Configmap Persistent Volume Claim -

# postgres master data configmap
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: postgres-master-configmap
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 5Gi
Postgres Master Data Persitent Volume -

# postgres master data volume
apiVersion: v1
kind: PersistentVolume
metadata:
name: postgres-data-master-pv
spec:
capacity:
storage: 5Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
hostPath:
path: /mnt/master
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- minikube # Node Name

WhatsApp: +8801715519132
Postgres Master Data Persistent Volume Claim -
# postgres master data
apiVersion: v1
metadata:
name: postgres-master-data-vol
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 5Gi
Postgres Slave Data Volume Claim -

# postgres slave data
apiVersion: v1
metadata:
name: postgres-slave-data-vol
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
# Create Volume(pv) and vVolume Claim (pvc)
kubectl apply -f postgres-volume.yaml
kubectl get pv -o wide
kubectl get pvc -o wide
POSTGRES MASTER CONFIGMAP ( postgres-master-configmap --from-file=config )
In this section we will focus on configure PostgreSQL Replication. To understand this you
have to understand the process of database replication configuration process. We have to
work with pg_hba.conf, postgresql.conf and configuration may very as per requirements.

WhatsApp: +8801715519132
Configuration pg_hba.conf
# TYPE DATABASE USER ADDRESS METHOD
local all all trust
# IPv4 local connections:
host all all 127.0.0.1/32 trust
# IPv6 local connections:
host all all ::1/128 trust
# Allow replication connections from localhost, by a user with the
# replication privilege.
local replication all trust
host replication all 127.0.0.1/32 trust
host replication all ::1/128 trust
host replication repuser 0.0.0.0/0 scram-sha-256
host all all all md5
Configuration postgresql.conf
listen_addresses = '*'
max_connections = 100
shared_buffers = 128MB
dynamic_shared_memory_type = posix
max_wal_size = 1GB
min_wal_size = 80MB
log_timezone = 'Etc/UTC'
datestyle = 'iso, mdy'
timezone = 'Etc/UTC'
lc_messages = 'en_US.utf8'
lc_monetary = 'en_US.utf8'
lc_numeric = 'en_US.utf8'
lc_time = 'en_US.utf8'
default_text_search_config = 'pg_catalog.english'
#-----------------------------------------------------------------------------
# CUSTOMIZED OPTIONS
#-----------------------------------------------------------------------------
# Add settings for extensions here
wal_level = replica
max_wal_senders = 2
max_replication_slots = 2
synchronous_commit = off
# Create Master ConfigMap (cm)
kubectl create cm postgres-master-configmap --from-file=config

WhatsApp: +8801715519132
POSTGRES MASTER StatefulSet (sts) - postgres-master-sts.yaml
Postgres StatefulSet Configuration to deploy Postgres Master Node PODs.
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: postgres-master
labels:
component: postgres-master
spec:
selector:
matchLabels:
serviceName: postgres-master
template:
metadata:
labels:
spec:
containers:
- name: postgres
image: postgres:11
command:
[
"sh",
"-c",
"docker-entrypoint.sh -c config_file=/var/config/postgresql.conf
-c hba_file=/var/config/pg_hba.conf",
]
# Readiness and Liveness Probe
readinessProbe:
exec:
command: [ "psql", "-Upostgres", "-dpostgres", "-c", "SELECT 1" ]
initialDelaySeconds: 10
timeoutSeconds: 10
livenessProbe:
exec:
timeoutSeconds: 10
ports:
- containerPort: 5432
envFrom:
- configMapRef:
7

WhatsApp: +8801715519132
- secretRef:
volumeMounts:
- mountPath: /var/lib/postgresql/data
name: postgres-master-data
- mountPath: /var/config
volumes:
- name: postgres-master-configmap
configMap:
- name: postgres-master-data
persistentVolumeClaim:
claimName: postgres-master-data-vol
# Create Master StatefulSet (sts)
kubectl apply -f postgres-master-sts.yaml
kubectl get sts -o wide
sleep 10
kubectl get pods -o wide
POSTGRES MASTER SERVICE - postgres-master-svc.yaml
apiVersion: v1
kind: Service
metadata:
name: postgres-master
spec:
selector:
type: NodePort
ports:
- name: postgres-master-port
port: 5432
targetPort: 5432
nodePort: 30032
protocol: TCP
# Create Master Service (svc)
kubectl apply -f postgres-master-svc.yaml
kubectl get svc -o wide

WhatsApp: +8801715519132
POSTGRES REPLICA USER (repuser) CREATION
After Deploying Postgres Master StatefulSet and Postgres Master Service, use the following
commands to create 'repuser' for replication configuration.
#Create Replica User
kubectl exec -it postgres-master-0 -- bash
su - postgres
psql
SET password_encryption = 'scram-sha-256';
CREATE ROLE repuser WITH REPLICATION PASSWORD 'postgres' LOGIN;
SELECT * FROM pg_create_physical_replication_slot('replica_1_slot');
SYNC MASTER DATA JOB - sync-master-data.yaml
apiVersion: batch/v1
kind: Job
metadata:
name: sync-master-data
spec:
template:
spec:
restartPolicy: OnFailure
containers:
- name: sync-master-data
image: postgres:11
command:
[
"sh",
"-c",
'PGPASSWORD="postgres" pg_basebackup -h postgres-master -D
/var/lib/slave-postgresql/data -U repuser -vP',
]
volumeMounts:
- mountPath: /var/lib/slave-postgresql/data
name: postgres-slave-data
volumes:
- name: postgres-slave-data
claimName: postgres-slave-data-vol

WhatsApp: +8801715519132
# Create Data Sync Job (job)
kubectl apply -f sync-master-data.yaml
kubectl get job -o wide
POSTGRES SLAVE CONFIGMAP ( postgres-slave-configmap --from-file=slave-config )
In this section we will focus on configure PostgreSQL Replication. To understand this you
have to understand the process of database replication configuration process. We have to
work with postgresql.conf, recovery.conf and configuration may very as per
requirements.
Configuration postgresql.conf
listen_addresses = '*'
max_connections = 100
shared_buffers = 128MB
dynamic_shared_memory_type = posix
max_wal_size = 1GB
min_wal_size = 80MB
log_timezone = 'Etc/UTC'
datestyle = 'iso, mdy'
timezone = 'Etc/UTC'
lc_messages = 'en_US.utf8'
lc_monetary = 'en_US.utf8'
lc_numeric = 'en_US.utf8'
lc_time = 'en_US.utf8'
default_text_search_config = 'pg_catalog.english'
#-----------------------------------------------------------------------------
# CUSTOMIZED OPTIONS
#-----------------------------------------------------------------------------
# Add settings for extensions here
hot_standby = on
wal_level = replica
max_wal_senders = 2
max_replication_slots = 2
synchronous_commit = off
Configuration recovery.conf
standby_mode = on
primary_conninfo = 'host=postgres-master port=5432 user=repuser
password=postgres application_name=r1'
primary_slot_name = 'replica_1_slot'
trigger_file = '/var/lib/postgresql/data/change_to_master'
10

WhatsApp: +8801715519132
# Create Slave ConfigMap (cm)
kubectl create cm postgres-slave-configmap --from-file=slave-config
POSTGRES SLAVE StatefulSet (sts) - postgres-slave-sts.yaml
Postgres StatefulSet Configuration to deploy Postgres Slave Node PODs.
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: postgres-slave
labels:
component: postgres-slave
spec:
selector:
matchLabels:
serviceName: postgres-slave
template:
metadata:
labels:
spec:
initContainers:
- name: busybox
image: busybox
command:
- sh
- -c
- "cp /var/config/postgresql.conf
/var/lib/postgresql/data/postgresql.conf && cp /var/config/recovery.conf
/var/lib/postgresql/data/recovery.conf"
volumeMounts:
- mountPath: /var/config/postgresql.conf
subPath: postgresql.conf
name: postgres-slave-configmap
- mountPath: /var/config/recovery.conf
11

WhatsApp: +8801715519132
subPath: recovery.conf
containers:
- name: postgres
image: postgres:11
# Readiness and Liveness Probe
readinessProbe:
exec:
timeoutSeconds: 10
livenessProbe:
exec:
timeoutSeconds: 10
ports:
- containerPort: 5432
envFrom:
- configMapRef:
- secretRef:
volumeMounts:
volumes:
- name: postgres-slave-configmap
configMap:
- name: postgres-slave-data
claimName: postgres-slave-data-vol
# Create Slave StatefulSet (sts)
kubectl apply -f postgres-slave-sts.yaml
kubectl get sts -o wide
sleep 10
kubectl get pods -o wide
12

WhatsApp: +8801715519132
ADDITIONAL HELPS COMMANDS
Some SQL Command to test the replication -

#==========SQL====================
kubectl exec -it postgres-master-0 -- psql -h localhost -U postgres -d postgres
CREATE TABLE test2 (id int not null, val text not null);
INSERT INTO test2 VALUES (1, 'foo1');
INSERT INTO test2 VALUES (2, 'bar2');
INSERT INTO test2 VALUES (3, 'zoo3');
kubectl exec -it postgres-slave-0 -- psql -h localhost -U postgres -d postgres

select * from test2;
Extra Help : Below commd lines will help you to expose the kubernetes Dashboad using
proxy
#Dashboard - Run this only if you want to expose the dashboard to outside of
the machine.
minikube dashboard &
kubectl proxy --address='0.0.0.0' --disable-filter=true &
http://127.0.0.1:35283/api/v1/namespaces/kubernetes-dashboard/services/http:kub
ernetes-dashboard:/proxy/
http://192.168.0.102:8001/api/v1/namespaces/kubernetes-dashboard/services/http:
kubernetes-dashboard:/proxy/
GITHUB PROJECT REPOSITORY
Repository Link: https://github.com/uzzal2k5/postgres-k8s
13

WhatsApp: +8801715519132
SPECIAL INSTRUCTIONS
Please follow commands from postgres-sts-deploy.sh sequentially. Otherwise

dependency will not meet and got error on sync-master-data and postgres-slave-0
container. If you encounter any error due to overlook the sequence, then cleanup / delete
everything specially statefulSet, Container and volumes, job and deploy again.
kubectl delete -f postgres-slave-sts.yaml

kubectl delete -f sync-master-data.yaml
kubectl delete -f postgres-master-svc.yaml
kubectl delete -f postgres-master-sts.yaml
kubectl delete -f postgres-volume.yaml
kubectl delete -f postgres-configmap.yaml
kubectl delete -f postgres-secret.yaml
kubectl delete pv --all
kubectl delete pvc --all
14

WhatsApp: +8801715519132

PostgreSQL Master Slave Replication On Kubernetes

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

PostgreSQL Master Slave Replication On Kubernetes

Uploaded by

Copyright:

Available Formats

PostgreSQL Master Slave Replication on Kubernetes

Master-slave replication is a process in database management systems (DBMS) where data

Author: Md Shafiqul Islam - [ Lead DevOps Architect ]

Files / Folder Structure -

POSTGRES CONFIGMAP - postgres-configmap.yaml

Author: Md Shafiqul Islam - [ Lead DevOps Architect ]

POSTGRES VOLUME - postgres-volume.yaml

Postgres Master Data Persistent Volume Configmap -

Author: Md Shafiqul Islam - [ Lead DevOps Architect ]

Postgres Master Data Configmap Persistent Volume Claim -

Postgres Master Data Persitent Volume -

Author: Md Shafiqul Islam - [ Lead DevOps Architect ]

Postgres Slave Data Volume Claim -

POSTGRES MASTER CONFIGMAP ( postgres-master-configmap --from-file=config )

Author: Md Shafiqul Islam - [ Lead DevOps Architect ]

Author: Md Shafiqul Islam - [ Lead DevOps Architect ]

Postgres StatefulSet Configuration to deploy Postgres Master Node PODs.

Author: Md Shafiqul Islam - [ Lead DevOps Architect ]

POSTGRES MASTER SERVICE - postgres-master-svc.yaml

Author: Md Shafiqul Islam - [ Lead DevOps Architect ]

SYNC MASTER DATA JOB - sync-master-data.yaml

Author: Md Shafiqul Islam - [ Lead DevOps Architect ]

POSTGRES SLAVE CONFIGMAP ( postgres-slave-configmap --from-file=slave-config )

Author: Md Shafiqul Islam - [ Lead DevOps Architect ]

POSTGRES SLAVE StatefulSet (sts) - postgres-slave-sts.yaml

Postgres StatefulSet Configuration to deploy Postgres Slave Node PODs.

Author: Md Shafiqul Islam - [ Lead DevOps Architect ]

Author: Md Shafiqul Islam - [ Lead DevOps Architect ]

Some SQL Command to test the replication -

kubectl exec -it postgres-master-0 -- psql -h localhost -U postgres -d postgres

kubectl exec -it postgres-slave-0 -- psql -h localhost -U postgres -d postgres

GITHUB PROJECT REPOSITORY

Repository Link: https://github.com/uzzal2k5/postgres-k8s

Author: Md Shafiqul Islam - [ Lead DevOps Architect ]

Please follow commands from postgres-sts-deploy.sh sequentially. Otherwise

kubectl delete -f postgres-slave-sts.yaml

Author: Md Shafiqul Islam - [ Lead DevOps Architect ]

You might also like