You are on page 1of 103

Using Cacti To Graph MySQLs Metrics

Kenny Gryp
kenny.gryp@percona.com Principal Consultant @ Percona

Collaborate 2011

Percona
MySQL/LAMP Consulting MySQL Support Percona Server (XtraDB) Percona XtraBackup InnoDB Recovery Toolkit, tcprstat (maatkit, innotop, aspersa, mysql-mmm, mysql-cactitemplates)

... http://www.percona.com http://www.mysqlperformanceblog.com


2

Topics
Why Graph? What is Cacti and mysql-cacti-templates? How Cacti works Installation Add Graphs Demo Graphs! Extras
3

Topics
Why Graph? What is Cacti and mysql-cacti-templates? How Cacti works Installation Add Graphs Demo Graphs! Extras
4

Why Graph?
Troubleshooting
-behavior over time -sudden spikes in graphs after application updates -traffic patterns -predictions

Capacity Planning

Not only for Operations, also for Development and


Management

Topics
Why Graph? What is Cacti and mysql-cacti-templates? How Cacti works Installation Add Graphs Demo Graphs! Extras
6

Cacti?
open source graphing solution uses RRDTool advanced graphs, scriptable web interface quite complex to add graphs adding servers is a manual process

mysql-cacti-templates?
collection of templates for cacti graph MySQL metrics Has a lot of non-MySQL Metrics too: IO stats,
apache, memcache, mongodb, java...

Topics
Why Graph? What is Cacti and mysql-cacti-templates? How Cacti works Installation Add Graphs Demo Graphs! Extras
9

How does Cacti work?


Database Server
/proc/ diskstats df mysqld sshd snmpd

Cacti
Poller
fetch
(cronjob which fetches data from servers)

what

MySQL
(contains cacti conguration)

Web Server
httpd sshd snmpd

stores

Apache
(serves web interface)

.rrd Files
(contains rrd data)

Web Browser

10

Topics
Why Graph? What is Cacti and mysql-cacti-templates? How Cacti works Installation Add Graphs Demo Graphs! Extras
11

Installation
Cacti Mysql-cacti-templates Configuring Servers

12

Installation
Cacti Mysql-cacti-templates Configuring Servers

13

Setting up Cacti: Requirements


httpd php, php-mysql, php-snmp mysql-server (server and client utilities) net-snmp (snmpget) rrdtool

14

Setting up Cacti: Installation


Install rpm/deb/... package or extract yourself Configure webserver: php5, cacti scripts Create cacti mysql user Import cacti database Edit configuration Setup cronjob for poller Finish installation using your webbrowser

15

Setup: Extract
Recommended is using apt/yum/... to install cacti if not:
cacti# cacti# cacti# cacti# cd /var/www/html/ # may vary tar xzvf cacti-0.8.7g.tar.gz ln -s cacti-0.8.7g cacti chown -R cactiuser rra/ log/

16

Setup: Webserver
Apache (may be something else) Enable php Set DirectoryIndex index.php
LoadModule php5_module modules/libphp5.so AddHandler php5-script .php AddType text/html .php DirectoryIndex index.php

17

Setup: MySQL
Create Cacti MySQL user:
cactimysql> GRANT ALL PRIVILEGES ON cacti.* TO cactiuser@localhost IDENTIFIED BY 'password';

Import Cacti Schema


cacti# mysql -e CREATE DATABASE cacti; cacti# mysql cacti < cacti.sql

Edit include/config.php
$database_type = "mysql"; $database_default = "cacti"; $database_hostname = "localhost"; $database_username = "cactiuser"; $database_password = "password"; 18

Setup: Poller
Add cronjob (/etc/cron.d/cacti)
*/5 * * * * cactiuser php /var/www/html/cacti/ poller.php > /dev/null 2>&1

19

Setup: UI

20

Setup: UI

21

Setup: UI

22

Setup: UI

23

Setup: UI

24

Setup: UI

25

Setup: UI

26

Installation
Cacti Mysql-cacti-templates Configuring Servers

27

download & installation


cacti# wget http://mysql-cactitemplates.googlecode.com/files/better-cactitemplates-1.1.8.tar.gz cacti# tar -xzf better-cacti-templates-1.1.8.tar.gz cacti# cd better-cacti-templates-1.1.8/ cacti# cp \ scripts/{ss_get_mysql_stats.php,ss_get_by_ssh.php}\ /var/www/html/cacti/scripts/

28

install: mysql script


cacti# vi /var/www/html/cacti/scripts/\ ss_get_mysql_stats.php ... $mysql_user = 'cactiuser'; $mysql_pass = 'password'; ...

29

install: generate ssh key


cacti# mkdir /etc/cacti cacti# ssh-keygen Generating public/private rsa key pair. Enter file in which to save the key (/root/.ssh/ id_rsa): /etc/cacti/id_rsa Enter passphrase (empty for no passphrase): Your identification has been saved in /etc/cacti/ id_rsa. Your public key has been saved in /etc/cacti/ id_rsa.pub. cacti# chown -R www-data: /etc/cacti/ {id_rsa,id_rsa.pub} cacti# chmod -R 0400 /etc/cacti/id_rsa
30

install: ssh script


cacti# vi /var/www/html/cacti/scripts/ ss_get_by_ssh.php ... $ssh_user= 'cacti'; # SSH username $ssh_port= 22; # SSH port $ssh_iden='-i /etc/cacti/id_rsa';# SSH identity ...

31

install: templates

32

install: templates

33

install: templates

34

Installation
Cacti Mysql-cacti-templates Configuring Servers

35

Configure Servers
configure net-snmpd add a cacti user in MySQL configure ssh access

36

Configure Servers: snmpd


Install snmpd
server# apt-get install snmpd

Make sure snmpd listens on the network and that a


server# vi /etc/snmp/snmpd.conf: agentAddress udp:161 rocommunity public server# /etc/init.d/snmpd restart

community password is set:

Verify on the cacti server if snmp is working remotely


cacti# snmpget -v2c -c public ubuntu-servertest iso.3.6.1.2.1.1.5.0 iso.3.6.1.2.1.1.5.0=STRING:"ubuntu-server-test"
37

Configure Servers: mysql user


Add cacti user
servermysql> GRANT SUPER, PROCESS ON *.* TO 'cactiuser'@'cactiserver' IDENTIFIED BY "password";

Verify if login works from cacti servers


cacti# mysql -h ubuntu-server-test -e "SELECT VERSION()" -u cactiuser -ppassword +-------------------+ | VERSION() | +-------------------+ | 5.1.49-1ubuntu8.1 | +-------------------+
38

Configure Servers: ssh access


Add shell user cacti server# useradd cacti
server# getent passwd cacti cacti:x:1001:1002::/home/cacti:/bin/sh

Copy the public key


server# server# EOF ssh-rsa EOF server# mkdir /home/cacti/.ssh/ cat > /home/cacti/.ssh/authorized_keys << AAA...r2E7z5 root@ubuntu-server-test chmod 0400 /home/cacti/.ssh/authorized_keys

cacti# ssh -i /etc/cacti/id_rsa cacti@ubuntu-servertest


39

Verify

Topics
Why Graph? What is Cacti and mysql-cacti-templates? How Cacti works Installation Add Graphs Demo Graphs! Extras
40

Adding Graphs

41

Adding Graphs

42

Adding Graphs

43

Adding Graphs
Apply wanted templates on the device Click on Create Graphs:
- MySQL & GNU/Linux templates

Select graphs to create and Add! Might ask some more questions
44

Adding Graphs

45

Adding Graphs
Assign ucd/net SNMP template to have:
-cpu usage/load -memory used/available -network statistics -free disk space

Different way of creating

46

Adding Graphs

47

Adding Graphs
What if I have multiple disk devices to monitor?
-Go to Create graphs for this host -Select graph to add in select menu -Create and fill in other device when asked

48

Adding Graphs: Graph Trees


Tree to browse through graphs By host/individual graph, mixed

49

Topics
Why Graph? What is Cacti and mysql-cacti-templates? How Cacti works Installation Add Graphs Demo Graphs! Extras
50

Topics
Why Graph? What is Cacti and mysql-cacti-templates? How Cacti works Installation Add Graphs Demo Graphs! Extras
51

MySQL Handlers

52

MySQL Handlers
Table scans, backup?

53

MySQL Handlers

54

MySQL Handlers

Take backup here


55

MySQL Connections

56

MySQL Connections
max_connections not reached yet

Why more incoming connections?


57

MySQL Replication

58

MySQL Replication
Replication never caught up

Replication was stopped


59

MySQL Replication

60

MySQL Replication
In 2 hours, almost 6900 seconds replication lag

61

MySQL Temporary Objects

62

MySQL Temporary Objects

63

keep an eye on: * temp tables * temp disk tables if it grows > query optimization

MySQL Select Types

64

MySQL Select Types


Why this sudden peak in rangescans?

65

MySQL Select Types

66

MySQL Select Types


Query behavior changed? Other explain plan chosen?

67

MySQL Sorts

68

MySQL Sorts

69

The peaks should be understood, not necessarily be removed as they might be really fast in terms of rows

MySQL Command Counters

70

MySQL Binary Logs

71

MySQL Binary Logs


expire_logs_days

72

Response Time Distribution


+----------------+-------+------------+ | time | count | total | +----------------+-------+------------| | 0.000001 | 0 | 0.000000 | | 0.000010 | 17 | 0.000094 | | 0.000100 | 4301 | 0.236555 | | 0.001000 | 1499 | 0.824450 | | 0.010000 | 14851 | 81.680502 | | 0.100000 | 8066 | 443.635693 | | 1.000000 | 0 | 0.000000 | | 10.000000 | 0 | 0.000000 | | 100.000000 | 1 | 55.937094 | | 1000.000000 | 0 | 0.000000 | | 10000.000000 | 0 | 0.000000 | 100000.000000 | 0 | 0.000000 | 73|

MySQL Query Response Time

74

MySQL Query Response Time

75

InnoDB Buffer Pool

76

InnoDB Buffer Pool

DB Restart Time to fill buffer


77

InnoDB Checkpoint Age

78

InnoDB Checkpoint Age


sudden increase in unflushed writes

db restart, no write flushing

79

InnoDB Buffer Pool Activity

80

InnoDB I/O

81

More read requests

InnoDB I/O
Increase in load

82

InnoDB Row Operations

83

InnoDB Row Operations

Good view on capacity of the system


84

85

more row reads, coming from disk

86

this spike in IO reads did not cause more disk reads

87

88

looking at amount of statements 89 does not always show load

CPU Usage

90

Memory

91

Disk Operations

92

Topics
Why Graph? What is Cacti and mysql-cacti-templates? How Cacti works Installation Add Graphs Demo Graphs! Extras
93

non-MySQL mysql-cacti-templates
Apache JMX Memcached Mongodb Nginx OpenVZ Redis Unix
94

non-MySQL mysql-cacti-templates

95

non-MySQL mysql-cacti-templates

96

non-MySQL mysql-cacti-templates

97

non-MySQL mysql-cacti-templates

98

Other Graphing solutions


Munin and munin-mysql OpenNMS and MySQL-SNMP Zabbix with appaloosa-zabbix-templates

99

Creating Your Own Graphs


Creating cacti graphs not trivial/portable mysql-cacti-templates to the rescue!
http://code.google.com/p/mysql-cacti-templates/ wiki/CreatingGraphs

100

Using Cacti To Graph MySQLs Metrics


Powerful Tool Easy to install Open source!

101

Using Cacti To Graph MySQLs Metrics


Cacti: http://www.cacti.net/ RRDtool: http://oss.oetiker.ch/rrdtool/ mysql-cacti-templates:

http://code.google.com/p/mysql-cacti-templates/ IRC: Freenode #percona Mailinglist: http://groups.google.com/group/percona-discussion http://www.percona.com http://www.mysqlperformanceblog.com


102

Percona Live MySQL Conference NYC May 26, 2011 http://www.percona.com/live/nyc-2011/

103