You are on page 1of 123

Implementing MySQL Security

Features
Ronald Bradford, Colin Charles
Percona Live Europe
Dublin 2017

#PerconaLive @RonaldBradford @bytebot


License
● Creative Commons BY-NC-SA 4.0
● https://creativecommons.org/licenses/by-nc-sa/4.0/legalcode

#PerconaLive @RonaldBradford @bytebot


About: Ronald Bradford
● Senior MySQL Database Consultant at Pythian
● Author/Blogger/Speaker
● Oracle ACE Director 2010 - present
● MySQL Community Member of the Year Award winner 09, 13
● Formally MySQL Inc 06-08, Oracle Corporation 96-99
● http://ronaldbradford.com/presentations/
● http://effectivemysql.com

#PerconaLive @RonaldBradford @bytebot


R
About: Colin Charles
● Chief Evangelist (in the CTO office), Percona Inc
● Founding team of MariaDB Server (2009-2016), previously at Monty Program
Ab, merged with SkySQL Ab, now MariaDB Corporation
● Formerly MySQL AB (exit: Sun Microsystems)
● Past lives include Fedora Project (FESCO), OpenOffice.org
● MySQL Community Contributor of the Year Award winner 2014
● http://bytebot.net/blog/

#PerconaLive @RonaldBradford @bytebot


Agenda
● A Security Primer ● Key management options
● Available ANSI and MySQL constructs ● Practical OS security (user accounts,
● Evolution of defaults in MySQL Versions sudo, iptables/ufw, SELinux/Appamor)
● Basic security hardening ● Why auditing is important
● Reviewing authentication modules ● Understanding the role of Clouds,
● Better/Stronger/Harder passwords Containers and Compliance
● Encryption options for variants Percona
Server, MariaDB, MySQL Enterprise
including LUKS, ezcrypt, Gazzang

#PerconaLive @RonaldBradford @bytebot


Let’s get setup
● Vagrant and Virtualbox is a good choice Vagrant/VirtualBox
○ https://github.com/ronaldbradford/mysql-security-tutorial on USB
● Docker

$ wget https://bit.ly/dockerhelper
$ source dockerhelper
$ docker_mysql | docker_percona | docker_mariadb

● You can also naturally just have a MySQL instance on your computer or cloud

#PerconaLive @RonaldBradford @bytebot


MySQL Variants and Versions… quick history lesson
● MySQL Community Edition
○ Won’t cover: 3.23, 4.0, 4.1, 5.0, 5.1
○ Will focus on: 5.5, 5.6, 5.7, with present development 8.0 version
● MySQL Enterprise Edition
● Percona Server for MySQL
○ 5.5, 5.6, 5.7
● MariaDB Server
○ Won’t cover: 5.1, 5.2, 5.3
○ 5.5, 10.0, 10.1, 10.2, with 10.3 as an alpha
● What we won’t cover: MySQL Cluster (NDBCLUSTER), Galera Cluster,
Group Replication/InnoDB Cluster, X Protocol/mysqlsh (33060)

#PerconaLive @RonaldBradford @bytebot


Structured Query Language (SQL)
● ISO/IEC 9075 (reviewed every 5 years) select @@global.sql_mode;
● SQL-86
● SQL-89 ● ANSI - come close to the SQL standard
● SQL-92 ● STRICT_TRANS_TABLES - If a value
● SQL:1999 could not be inserted as given into a
● SQL:2003 transactional table, abort the statement.
● SQL:2006 ● TRADITIONAL - “give an error instead of a
● SQL:2008 warning” when inserting an incorrect value
● SQL:2011 into a column.
● SQL:2016 ● https://dev.mysql.com/doc/refman/5.7/en/s
ql-mode.html

Deprecated - MariaDB Server has


NO_AUTO_CREATE_USER but MySQL 5.7
has this in standard sql_mode
#PerconaLive @RonaldBradford @bytebot
MySQL Security by version
● GRANT (3.23) ● SHOW CREATE USER (5.7)
● REVOKE (3.23) ● CREATE ROLE (8.0)
● SET PASSWORD (3.23) ● DROP ROLE (8.0)
● SHOW GRANTS (3.23) ● SET ROLE (8.0)
● DROP USER (4.1) ● SET DEFAULT ROLE (8.0)
● SHOW PRIVILEGES (4.1) ● N/B: ROLES came to MariaDB Server in
● CREATE USER (5.0) 10.0, and the DEFAULT ROLE came in
● RENAME USER (5.0) 10.1
● ALTER USER (5.6)

#PerconaLive @RonaldBradford @bytebot


mysql.users table
● host
● user
● password (removed in 5.7; still present in MariaDB)
● plugin (5.5)
● authentication_string (5.5)
● password_expired (5.6)
● account_locked (5.7)
● Create_role_priv (8.0)
● Drop_role_priv (8.0)

Comparing mysql.user between MariaDB Server 10.2 and MySQL 5.7

#PerconaLive @RonaldBradford @bytebot


Key Security Features by Version
● 5.1 - McAfee Audit plugin ● 5.7 - grep for root password on installation,
● 5.5 - pluggable authentication (MariaDB 5.2 password expiry every ‘n’ days, user accounts
backport), proxy users, changes in mysql.user can be locked/unlocked, mysql_ssl_rsa_setup,
table, client password warning; Enterprise mysql.user.password removed, super_read_only,
provided Audit and PAM authentication (present at rest tablespace encryption
again in Percona Server for MySQL and MariaDB ● 8.0 - roles + mysql.user changes
Server) ● MariaDB 10.0 - roles, userstats
● 5.6 - encrypted client credentials ● MariaDB 10.1 - default roles, at rest
(mysql_config_editor), sha256_password, table/tablespace encryption,
password expiry, simple_password_check,
VALIDATE_PASSWORD_STRENGTH(), cracklib_password_check, AWS Key
--random-passwords (optional random on install), Management plugin
mysql.user password_expired column; Enterprise ● MariaDB 10.2 - user limits, ed25519 auth
Firewall ● Percona 5.5 - extended SHOW GRANTS, utility
user, userstats
● Percona 5.6 - super_read_only

#PerconaLive @RonaldBradford @bytebot


Passwords

#PerconaLive @RonaldBradford @bytebot


R
Installation Default Passwords
● 'root' user
○ Pre 5.7 no password
○ 5.7 expired random password
● Anonymous users
○ Removed in 5.7

#PerconaLive @RonaldBradford @bytebot


R
MySQL Native Password Hash
mysql> SELECT PASSWORD('test123') AS passwd;
+-------------------------------------------+
| passwd |
+-------------------------------------------+
| *676243218923905CF94CB52A3C9D3EB30CE8E20D |
+-------------------------------------------+
1 row in set (0.00 sec)

mysql> SELECT CONCAT('*',UPPER( SHA1(UNHEX(SHA1('test123'))))) AS gen_passwd;


+-------------------------------------------+
| gen_passwd |
+-------------------------------------------+
| *676243218923905CF94CB52A3C9D3EB30CE8E20D |
+-------------------------------------------+
1 row in set (0.00 sec)

#PerconaLive @RonaldBradford @bytebot


R
What is a native password?
mysql> SELECT CONCAT('*',UPPER(SHA1(UNHEX(SHA1('test123'))))) AS gen_passwd;
+-------------------------------------------+
| gen_passwd |
+-------------------------------------------+
| *676243218923905CF94CB52A3C9D3EB30CE8E20D |
+-------------------------------------------+
1 row in set (0.00 sec)

$ echo -n "test123" | sha1sum | cut -c1-40 | xxd -p -r | sha1sum | cut -c1-40 | tr '[a-z]'
'[A-Z]'
676243218923905CF94CB52A3C9D3EB30CE8E20D

$ sudo yum install -y vim-common # xxd

#PerconaLive @RonaldBradford @bytebot


R
mysql.user table on the filesystem
$ cd /var/lib/mysql/mysql/
$ strings user.MYD
localhost
root
mysql_native_password
...

mysql> SET PASSWORD = PASSWORD('test123');

$ strings user.MYD
localhost
root*676243218923905CF94CB52A3C9D3EB30CE8E20D
...

#PerconaLive @RonaldBradford @bytebot


R
What is SHA?
● The Secure Hash Algorithms (SHA) are a family of cryptographic hash
functions
● SHA/SHA-1 is 160 bits == 20 bytes == 40 hexadecimal characters
SHA2 is SHA-224, SHA-256, SHA-384, and SHA-512

https://en.wikipedia.org/wiki/Secure_Hash_Algorithms

#PerconaLive @RonaldBradford @bytebot


R
How are passwords stored in MySQL (5.5)
mysql55 >SELECT /* 5.5 */ host, user, password, plugin, authentication_string FROM mysql.user;
+-----------+------+----------+--------+-----------------------+
| host | user | password | plugin | authentication_string |
+-----------+------+----------+--------+-----------------------+
| localhost | root | | | |
| mysql55 | root | | | |
| 127.0.0.1 | root | | | |
| ::1 | root | | | |
| localhost | | | | NULL |
| mysql55 | | | | NULL |
+-----------+------+----------+--------+-----------------------+
6 rows in set (0.00 sec)

plugin
unspecified

#PerconaLive @RonaldBradford @bytebot


R
How are passwords stored in MySQL (5.6)
mysql56 >SELECT /* 5.6 */ host, user, password, plugin, authentication_string, password_expired FROM mysql.user;
+-----------+------+----------+-----------------------+-----------------------+------------------+
| host | user | password | plugin | authentication_string | password_expired |
+-----------+------+----------+-----------------------+-----------------------+------------------+
| localhost | root | | mysql_native_password | | N |
| mysql56 | root | | mysql_native_password | | N |
| 127.0.0.1 | root | | mysql_native_password | | N |
| ::1 | root | | mysql_native_password | | N |
| localhost | | | mysql_native_password | NULL | N |
| mysql56 | | | mysql_native_password | NULL | N |
+-----------+------+----------+-----------------------+-----------------------+------------------+
6 rows in set (0.00 sec)

plugin specified

#PerconaLive @RonaldBradford @bytebot


R
How are passwords stored in MySQL (5.7)
mysql57 >SELECT /* 5.7 */ host, user, plugin, authentication_string, password_expired,
password_last_changed, password_lifetime, account_locked FROM mysql.user;
+-----------+---------------+-----------------------+-------------------------------------------+---------------
---+-----------------------+-------------------+----------------+
| host | user | plugin | authentication_string |
password_expired | password_last_changed | password_lifetime | account_locked |
+-----------+---------------+-----------------------+-------------------------------------------+---------------
---+-----------------------+-------------------+----------------+
| localhost | root | mysql_native_password | *E89C1DBB80A00976B61D19025C3081E4B190D8BE | N
| 2017-09-03 18:45:43 | NULL | N |
| localhost | mysql.session | mysql_native_password | *THISISNOTAVALIDPASSWORDTHATCANBEUSEDHERE | N
| 2017-09-03 18:42:33 | NULL | Y |
| localhost | mysql.sys | mysql_native_password | *THISISNOTAVALIDPASSWORDTHATCANBEUSEDHERE | N
| 2017-09-03 18:42:33 | NULL | Y |
+-----------+---------------+-----------------------+-------------------------------------------+---------------
---+-----------------------+-------------------+----------------+
3 rows in set (0.01 sec)
Password stored in
authentication_string

#PerconaLive @RonaldBradford @bytebot


R
Minimum password policy (5.7)
mysql57 > ALTER USER USER() IDENTIFIED BY 'percona';
ERROR 1819 (HY000): Your password does not satisfy the current policy requirements
mysql57 > ALTER USER USER() IDENTIFIED BY 'percona123';
ERROR 1819 (HY000): Your password does not satisfy the current policy requirements
mysql57 > ALTER USER USER() IDENTIFIED BY 'Percona123';
ERROR 1819 (HY000): Your password does not satisfy the current policy requirements
mysql57 > ALTER USER USER() IDENTIFIED BY 'Percona.123';
Query OK, 0 rows affected (0.00 sec)
mysql57 >SELECT @@validate_password_policy;
+----------------------------+
| @@validate_password_policy |
+----------------------------+
| MEDIUM |
+----------------------------+

https://dev.mysql.com/doc/refman/5.7/en/validate-password-options-variables.html#sysvar_validate_password_policy

#PerconaLive @RonaldBradford @bytebot


R
VALIDATE_PASSWORD_STRENGTH()
mysql> SELECT VALIDATE_PASSWORD_STRENGTH('percona');
+---------------------------------------+
| validate_password_strength('percona') |
+---------------------------------------+
| 25 |
+---------------------------------------+

mysql> SELECT VALIDATE_PASSWORD_STRENGTH('percona123');


+------------------------------------------+
| 50 |
+------------------------------------------+

mysql> SELECT VALIDATE_PASSWORD_STRENGTH('Percona123');


+------------------------------------------+
| 50 |
+------------------------------------------+

mysql> SELECT VALIDATE_PASSWORD_STRENGTH('Percona.123');


+-------------------------------------------+
| 100 |
+-------------------------------------------+

https://dev.mysql.com/doc/refman/5.7/en/encryption-functions.html#function_validate-password-strength

#PerconaLive @RonaldBradford @bytebot


R
VALIDATE_PASSWORD_STRENGTH()
mysql> SHOW GLOBAL VARIABLES LIKE 'validate%';
+--------------------------------------+--------+
| Variable_name | Value |
+--------------------------------------+--------+
| validate_password_check_user_name | OFF |
| validate_password_dictionary_file | |
| validate_password_length | 8 |
| validate_password_mixed_case_count | 1 |
| validate_password_number_count | 1 |
| validate_password_policy | MEDIUM |
| validate_password_special_char_count | 1 |
+--------------------------------------+--------+
7 rows in set (0.01 sec)

https://dev.mysql.com/doc/refman/5.7/en/validate-password-options-variables.html

#PerconaLive @RonaldBradford @bytebot


R
mysql_native_password format (deprecated in 5.7)
mysql56 > SELECT PASSWORD('test123') AS pwd;
+-------------------------------------------+
| pwd |
+-------------------------------------------+
| *676243218923905CF94CB52A3C9D3EB30CE8E20D |
+-------------------------------------------+
1 row in set (0.00 sec)

mysql57 > SELECT PASSWORD('test123') AS pwd;


+-------------------------------------------+
| pwd |
+-------------------------------------------+
| *676243218923905CF94CB52A3C9D3EB30CE8E20D |
+-------------------------------------------+
1 row in set, 1 warning (0.00 sec)

https://dev.mysql.com/doc/refman/5.7/en/server-system-variables.html#sysvar_old_passwords
https://dev.mysql.com/doc/refman/5.7/en/password-hashing.html

#PerconaLive @RonaldBradford @bytebot


R
Anonymous users
mysql55 $ mysql -e "SELECT VERSION(), USER(), CURRENT_USER()"
+-----------+-------------------+----------------+
| VERSION() | USER() | CURRENT_USER() |
+-----------+-------------------+----------------+
| 5.5.57 | vagrant@localhost | @localhost |
+-----------+-------------------+----------------+

mysql56 $ mysql -e "SELECT VERSION(), USER(), CURRENT_USER()"


+-----------+-------------------+----------------+
| VERSION() | USER() | CURRENT_USER() |
+-----------+-------------------+----------------+
| 5.6.37 | vagrant@localhost | @localhost |
+-----------+-------------------+----------------+

mysql57 $ mysql -e "SELECT VERSION(), USER(), CURRENT_USER()"


ERROR 1045 (28000): Access denied for user 'vagrant'@'localhost' (using password: NO)

#PerconaLive @RonaldBradford @bytebot


R
What can anonymous users do?
mysql55 >SHOW SCHEMAS;
+--------------------+
| Database |
+--------------------+
| information_schema |
| test |
+--------------------+

mysql56 >SHOW SCHEMAS;


+--------------------+
| information_schema |
+--------------------+

mysql55 >SELECT table_schema, table_name FROM information_schema.tables;


...

mysql55 >SELECT host,user,password FROM mysql.user;


ERROR 1142 (42000): SELECT command denied to user ''@'localhost' for table 'user'

#PerconaLive @RonaldBradford @bytebot


R
What can anonymous users do?
mysql55 >USE test;
mysql55 >CREATE TABLE t1(i1 INT NOT NULL AUTO_INCREMENT PRIMARY KEY, v1 VARCHAR(100) NOT
NULL);
mysql55 >INSERT INTO t1(i1, v1) VALUES (1, REPEAT('abcde',20));
mysql55 >INSERT INTO t1(i1, v1) SELECT NULL, a.v1 FROM t1 a, t1 b, t1 c;
Query OK, 1 row affected (0.00 sec)
mysql55 >INSERT INTO t1(i1, v1) SELECT NULL, a.v1 FROM t1 a, t1 b, t1 c;
Query OK, 8 rows affected (0.00 sec)
mysql55 >INSERT INTO t1(i1, v1) SELECT NULL, a.v1 FROM t1 a, t1 b, t1 c;
Query OK, 1000 rows affected (0.02 sec)
# Does it return in your VM, or fill your host disk?

Be Destructive

#PerconaLive @RonaldBradford @bytebot


R
Anonymous users in mysql.user
mysql56 >SELECT host,user,password FROM mysql.user WHERE user='';
+-----------+------+----------+
| host | user | password |
+-----------+------+----------+
| localhost | | |
| mysql56 | | |
+-----------+------+----------+
2 rows in set (0.00 sec)

mysql56 >DELETE FROM mysql.user WHERE user='';


mysql56 >FLUSH PRIVILEGES;

#PerconaLive @RonaldBradford @bytebot


R
Security Hardening 101
● Set 'root' password
● Remove anonymous users
● Remove 'test' schema

This is all taken care of with mysql_secure_installation

#PerconaLive @RonaldBradford @bytebot


R
mysql_secure_installation
$ mysql_secure_installation
5.7 Functionality
VALIDATE PASSWORD PLUGIN can be used to test passwords
and improve security. It checks the strength of password
and allows the users to set only those passwords which are
secure enough. Would you like to setup VALIDATE PASSWORD plugin?

There are three levels of password validation policy:

LOW Length >= 8


MEDIUM Length >= 8, numeric, mixed case, and special characters
STRONG Length >= 8, numeric, mixed case, special characters and dictionary
file

Please enter 0 = LOW, 1 = MEDIUM and 2 = STRONG:


...
Estimated strength of the password: 25
Do you wish to continue with the password provided?(Press y|Y for Yes, any other key for No)
...

#PerconaLive @RonaldBradford @bytebot


R
Command Line options (non-interactive)
read MYSQL_USER
read -s MYSQL_PWD

mysql -u${MYSQL_USER} -p${MYSQL_PWD} ps rewrite

ps -ef | grep mysql


.... mysql -udemo -px xxxxxxx

mysql # using $HOME/.my.cnf


mysql --defaults-file=/path/to/.my.cnf

● What about gh-ost, pt-osc, other pt- tools etc

https://dev.mysql.com/doc/refman/5.6/en/mysql-logging.html

#PerconaLive @RonaldBradford @bytebot


R
Bad habits Do not do this

$ (mysql -uinsecure -pTest.123 -e "SELECT SLEEP(3)" > /dev/null &); ps -ef | grep mysql
vagrant 2923 1 0 00:08 pts/0 00:00:00 mysql -uinsecure -px xxxxxxxxx -e SELECT SLEEP(3)

$ (pt-show-grants -uinsecure -pTest.123 > /dev/null &); ps -ef | grep pt


vagrant 2902 1 0 00:04 pts/0 00:00:00 perl /usr/bin/pt-show-grants -uinsecure
-pTest.123

$ (gh-ost --user=insecure --password=Test.123 &); ps -ef | grep gh


vagrant 2983 1 0 00:29 pts/0 00:00:00 gh-ost --user=insecure --password=Test.123

$ (xtrabackup -uinsecure -pTest.123 --backup >/dev/null &); ps -ef | grep xtra


vagrant 2917 1 0 00:08 pts/0 00:00:00 xtrabackup -uinsecure -px xxxxxxxxx --backup

#PerconaLive @RonaldBradford @bytebot


R
MySQL Config Editor
$ mysql_secure_installation
$ mysql -uroot -p -e "CREATE USER demo@localhost IDENTIFIED BY 'passw0rd1';"
$ echo "[client]
user=demo
password=passw0rd1" > $HOME/.my.cnf
$ mysql -e "SELECT USER()" Since 5.6
$ rm $HOME/.my.cnf
$ mysql -e "SELECT USER()"

$ mysql_config_editor set --login-path=client --host=localhost --user=demo --password


$ ls -al $HOME/.mylogin.cnf
$ cat $HOME/.mylogin.cnf
$ mysql_config_editor print

$ mysql -e "SELECT USER()"


$ mysqldump ....

#PerconaLive @RonaldBradford @bytebot


R
Password Improvements

#PerconaLive @RonaldBradford @bytebot


MySQL 5.6 improvements
● Password expiry - ALTER USER 'foo'@'localhost' PASSWORD EXPIRE;
● Password validation plugin - VALIDATE_PASSWORD_STRENGTH()
● mysql_config_editor - store authentication credentials in an encrypted
login path file named .mylogin.cnf
○ http://dev.mysql.com/doc/refman/5.6/en/mysql-config-editor.html

● Random ‘root’ password on install


○ mysql_install_db —random-passwords
○ cat $HOME/.mysql_secret

#PerconaLive @RonaldBradford @bytebot


PASSWORD EXPIRE
$ export MYSQL_PS1="\u@\h [\d]> "

$ mysql -uroot -p
root@localhost [(none)]> CREATE USER demo IDENTIFIED BY 'passw0rd1';
$ mysql -udemo -p #passw0rd1

$ mysql -uroot
root@localhost [(none)]> ALTER USER demo PASSWORD EXPIRE;

$ mysql -udemo -p #passw0rd1


demo@localhost [(none)]> # No issue connecting
demo@localhost [(none)]> USE test;
ERROR 1820 (HY000): You must reset your password using ALTER USER statement before
executing this statement.

# can reset password to current value


demo@localhost [(none)]> ALTER USER demo IDENTIFIED BY 'passw0rd1';

#PerconaLive @RonaldBradford @bytebot


MySQL 5.7 improvements
● Improved password expiry — automatic password expiration available, so set
default_password_lifetime in my.cnf
● You can also require password to be changed every n-days
○ ALTER USER foo@localhost PASSWORD EXPIRE INTERVAL n DAY;

● PASSWORD EXPIRE DEFAULT | NEVER options


● There is also account locking/unlocking now
○ ALTER USER foo@host ACCOUNT LOCK | UNLOCK;

https://dev.mysql.com/doc/refman/5.7/en/alter-user.html

#PerconaLive @RonaldBradford @bytebot


MariaDB Server passwords
● Password validation plugin
○ https://mariadb.com/kb/en/mariadb/development/mariadb-internals-documentation/password-v
alidation/

● simple_password_check password validation plugin


○ can enforce a minimum password length and guarantee that a password contains at least a
specified number of uppercase and lowercase letters, digits, and punctuation characters.

● cracklib_password_check password validation plugin


○ Allows passwords that are strong enough to pass CrackLib test. This is the same test that
pam_cracklib.so does
○ SET PASSWORD FOR 'bob'@'%.loc.gov' = PASSWORD('abc');

#PerconaLive @RonaldBradford @bytebot


Authentication in MySQL / MariaDB Server
● auth_socket - Authenticates against the Unix socket file, using so_peercred
● sha256_password - default-authentication-plugin=sha256_password,
passwords never exposed as cleartext when connecting; SSL or RSA auth
● ed25519 - Elliptic Curve Digital Signature Algorithm, same as OpenSSH
● Kerberos/GSSAPI/SSPI - User principals: <username>@<KERBEROS
REALM>
● Active Directory (Enterprise only)
● mysql_no_login ( MySQL 5.7 ) - prevents all client connections to an account
that uses it
#PerconaLive @RonaldBradford @bytebot
Users and Privileges

#PerconaLive @RonaldBradford @bytebot


Managing users
mysql> CREATE USER demo@localhost;
Query OK, 0 rows affected (0.00 sec)

$ mysql -udemo -e "SELECT USER(), CURRENT_USER()"


+----------------+----------------+
| USER() | CURRENT_USER() |
+----------------+----------------+
| demo@localhost | demo@localhost |
+----------------+----------------+

mysql> DROP USER demo@localhost;


Query OK, 0 rows affected (0.00 sec)

mysql57 >CREATE USER demo@localhost;


ERROR 1819 (HY000): Your password does not satisfy the current policy requirements

#PerconaLive @RonaldBradford @bytebot


R
User Connection Errors
$ mysql -udemo
ERROR 1045 (28000): Access denied for user 'demo'@'localhost' (using password: NO)

$ mysql -udemo -pxxx


mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 1045 (28000): Access denied for user 'demo'@'localhost' (using password: YES)

$ tail -2 /var/log/mysqld.log
2017-09-06T00:39:24.908224Z 16 [Note] Access denied for user 'demo'@'localhost' (using
password: NO)
2017-09-06T00:39:37.423691Z 17 [Note] Access denied for user 'demo'@'localhost' (using
password: YES)

#PerconaLive @RonaldBradford @bytebot


User Connection Errors via userstats (Percona
Server / MariaDB Server)
percona56 >set global userstat=ON;
percona56 >SELECT user,total_connections,denied_connections,total_ssl_connections FROM
information_schema.user_statistics;
percona56 >select host,user from mysql.user;

$ mysql -udemo -p
...

$ mysql -uroot -p
percona56 >SELECT user,total_connections,denied_connections,total_ssl_connections FROM
information_schema.user_statistics;
+------+-------------------+--------------------+-----------------------+
| user | total_connections | denied_connections | total_ssl_connections |
+------+-------------------+--------------------+-----------------------+
| demo | 1 | 0 | 0 |
| root | 3 | 0 | 0 |
+------+-------------------+--------------------+-----------------------+
2 rows in set (0.00 sec)

#PerconaLive @RonaldBradford @bytebot


SHOW CREATE USER (5.7)
mysql57 >SHOW CREATE USER root@localhost\G
*************************** 1. row ***************************
… CREATE USER 'root'@'localhost' IDENTIFIED WITH 'mysql_native_password' AS
'*E89C1DBB80A00976B61D19025C3081E4B190D8BE' REQUIRE NONE PASSWORD EXPIRE DEFAULT ACCOUNT UNLOCK

SHOW CREATE USER (Privileges needed)


host >select USER();
+-----------------------+
| USER() |
+-----------------------+
| external@192.168.42.1 |
+-----------------------+

host >SHOW CREATE USER external@'%';


...
| CREATE USER 'external'@'%' IDENTIFIED WITH 'mysql_native_password' AS
'*E89C1DBB80A00976B61D19025C3081E4B190D8BE' REQUIRE NONE PASSWORD EXPIRE DEFAULT ACCOUNT UNLOCK |

host >SHOW CREATE USER external@localhost;


ERROR 1044 (42000): Access denied for user 'external'@'%' to database 'mysql'

#PerconaLive @RonaldBradford @bytebot


MySQL privileges
mysql -e "SHOW GRANTS"
mysql -uroot -e "SHOW GRANTS"
mysql57 >SHOW PRIVILEGES;
--------------------------------------+
| Privilege | Context | Comment |
+-------------------------+---------------------------------------+-------------------------------------------------------+
| Alter | Tables | To alter the table |
| Alter routine | Functions,Procedures | To alter or drop stored functions/procedures |
| Create | Databases,Tables,Indexes | To create new databases and tables |
| Create routine | Databases | To use CREATE FUNCTION/PROCEDURE |
| Create temporary tables | Databases | To use CREATE TEMPORARY TABLE |
| Create view | Tables | To create new views |
| Create user | Server Admin | To create new users

#PerconaLive @RonaldBradford @bytebot


MySQL 8.0 additional privileges (8.0.2 DMR)
Create role Server Admin To create new roles
Drop role Server Admin To drop roles
SET_USER_ID Server Admin
ROLE_ADMIN Server Admin
REPLICATION_SLAVE_ADMIN Server Admin
XA_RECOVER_ADMIN Server Admin
GROUP_REPLICATION_ADMIN Server Admin
SYSTEM_VARIABLES_ADMIN Server Admin
PERSIST_RO_VARIABLES_ADMIN Server Admin
ENCRYPTION_KEY_ADMIN Server Admin
CONNECTION_ADMIN Server Admin
BINLOG_ADMIN Server Admin

#PerconaLive @RonaldBradford @bytebot


Sane user creation practices
● Use a least privileges model
● Segregate responsibilities of access
○ application (not DDL)
○ Read_only
○ DBA
● Use host as a means of limitation combined with OS access
● GRANT/REVOKE is not enough to return privileges to a defined state.
● DROP USER/CREATE USER/GRANT is necessary

#PerconaLive @RonaldBradford @bytebot


Why being SUPER is bad (GRANT ALL ON *.*)
● Bypasses read_only (why we need super_read_only)
● Bypasses init_connect
● Can disable binary logging
● Can change dynamic configuration
● Takes the reserved connection

http://ronaldbradford.com/blog/why-grant-all-is-bad-2010-08-06/

http://effectivemysql.com/presentation/mysql-idiosyncrasies-that-bite/

#PerconaLive @RonaldBradford @bytebot


Create users for specific purposes
Be specific:

● 'grafana' not 'monitoring'


● 'ghost'
● 'backup'
● Individual DBA user accounts

#PerconaLive @RonaldBradford @bytebot


Bypassing all privileges
--skip-grant-tables

$ more /etc/my.cnf

[mysqld]
skip-grant-tables

$ sudo systemctl restart mysqld.service

$ mysql -uroot # No password

https://dev.mysql.com/doc/refman/5.7/en/server-options.html#option_mysqld_sk
ip-grant-tables

#PerconaLive @RonaldBradford @bytebot


Bypassing all privileges
--init-file

$ more /etc/my.cnf

[mysqld]
init-file=/tmp/password.sql

$ cat /tmp/password.sql
GRANT ALL PRIVILEGES ON *.* TO root@localhost IDENTIFIED BY '<clear-text-password>'' WITH
GRANT OPTION;

$ sudo systemctl restart mysqld.service

$ mysql -uroot -p
See also: http://code.openark.org/blog/mysql/dangers-of-skip-grant-tables

#PerconaLive @RonaldBradford @bytebot


Bypassing all privileges (replacing the user.MYD file)
$ cd /tmp
$ echo "[mysqld]
datadir=/tmp" > my.cnf

$ scripts/mysql_install_db --defaults-file=/tmp/my.cnf
$ ls -l /tmp/mysql/user.*
-rw-rw---- 1 320 Feb 13 11:27 /tmp/mysql/user.MYD
-rw-rw---- 1 2048 Feb 13 11:27 /tmp/mysql/user.MYI
-rw-rw---- 1 10630 Feb 13 11:27 /tmp/mysql/user.frm

#PerconaLive @RonaldBradford @bytebot


MySQL Socket File (Localhost)
$ ps -ef | grep mysql
root 3016 1 0 Sep04 ? 00:00:00 /bin/sh /usr/bin/mysqld_safe
mysql 3169 3016 0 Sep04 ? 00:05:07 /usr/sbin/mysqld
--basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin
--user=mysql --log-error=/var/log/mysqld.log
--pid-file=/var/run/mysqld/mysqld.pid --socket=/var/lib/mysql/mysql.sock

$ grep socket /etc/my.cnf


socket=/var/lib/mysql/mysql.sock

#PerconaLive @RonaldBradford @bytebot


MySQL TCP/IP (external access)
$ grep 3306 /etc/services
mysql 3306/tcp # MySQL
mysql 3306/udp # MySQL
mysql57 > SELECT @@bind_address;
+----------------+
| @@bind_address |
+----------------+
| * |
+----------------+

mysql57$ netstat -an | grep 3306


Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
...
tcp6 0 0 :::3306 :::* LISTEN

#PerconaLive @RonaldBradford @bytebot


R
Using an OS firewall
host$ mysql -uexternal -p -h192.168.42.17
...

mysql$ sudo systemctl start firewalld

host$ mysql -uexternal -p -h192.168.42.17


Enter password:
ERROR 2003 (HY000): Can't connect to MySQL server on '192.168.42.17' (113)

#PerconaLive @RonaldBradford @bytebot


Enable access via the OS firewall
mysql$ sudo firewall-cmd --permanent --add-service=mysql
mysql$ sudo firewall-cmd --permanent --list-all
public
target: default
icmp-block-inversion: no
interfaces:
sources:
services: dhcpv6-client mysql ssh
...

mysql$ sudo firewall-cmd --reload

host$ mysql -uexternal -p -h192.168.42.17


...

mysql$ sudo systemctl stop firewalld

#PerconaLive @RonaldBradford @bytebot


SSL

#PerconaLive @RonaldBradford @bytebot


Secure Communications
● SSL for replication
● SSL for client connections
● SSL for admin connections
● Encryption on the wire

https://dev.mysql.com/doc/refman/5.6/en/secure-connections.html
https://dev.mysql.com/doc/refman/5.7/en/secure-connections.html

#PerconaLive @RonaldBradford @bytebot


Default client connection traffic (5.6)
mysql56$ sudo ngrep -d eth1 -wi -P ' ' -W single -l port 3306
interface: eth1 (192.168.42.0/255.255.255.0)
filter: ( port 3306 ) and ((ip || ip6) || (vlan && (ip || ip6)))

host$ mysql -uexternal -p -h192.168.42.16

host> select 'unencrypted';

#
T 192.168.42.1:47634 -> 192.168.42.16:3306 [AP] select 'unencrypted'
#
T 192.168.42.16:3306 -> 192.168.42.1:47634 [AP] ! def unencrypted ! !
unencrypted

https://wiki.christophchamp.com/index.php?title=Ngrep
http://infoheap.com/ngrep-quick-start-guide/

#PerconaLive @RonaldBradford @bytebot


Default client connection traffic (5.7)
mysql57$ sudo ngrep -d eth1 -wi -P ' ' -W single -l port 3306
interface: eth1 (192.168.42.0/255.255.255.0)
filter: ( port 3306 ) and ((ip || ip6) || (vlan && (ip || ip6)))

host$ mysql -uexternal -p -h192.168.42.17

host> select 'encrypted';

T 192.168.42.1:36781 -> 192.168.42.17:3306 [AP] @ F l d iVr H b ^ s t Z


( 2d " ? | )
#
T 192.168.42.17:3306 -> 192.168.42.1:36781 [AP] p% s` 3u5!%P] v= r # x E a y
'! )Z 8 Js z. \t (r H@ 0 2 5k\ < M @)E& b q|q@ h

#PerconaLive @RonaldBradford @bytebot


5.7 non-SSL client connection traffic
host$ > mysql -uexternal -p -h192.168.42.17 --ssl=0

host >select '-ssl=0 unencrypted';

T 192.168.42.1:36785 -> 192.168.42.17:3306 [AP] select '-ssl=0 unencrypted'


#
T 192.168.42.17:3306 -> 192.168.42.1:36785 [AP] ' def -ssl=0 unencrypted ! 3
-ssl=0 unencrypted

#PerconaLive @RonaldBradford @bytebot


SSL system variables
mysql57 > SHOW GLOBAL VARIABLES LIKE mysql56 >SHOW VARIABLES LIKE '%ssl%';
'%ssl%'; +---------------+----------+
+---------------+-----------------+ | Variable_name | Value |
| Variable_name | Value | +---------------+----------+
+---------------+-----------------+ | have_openssl | DISABLED |
| have_openssl | YES | | have_ssl | DISABLED |
| have_ssl | YES | | ssl_ca | |
| ssl_ca | ca.pem | | ssl_capath | |
| ssl_capath | | | ssl_cert | |
| ssl_cert | server-cert.pem | | ssl_cipher | |
| ssl_cipher | | | ssl_crl | |
| ssl_crl | | | ssl_crlpath | |
| ssl_crlpath | | | ssl_key | |
| ssl_key | server-key.pem | +---------------+----------+
+---------------+-----------------+ 9 rows in set (0.02 sec)
9 rows in set (0.00 sec)

#PerconaLive @RonaldBradford @bytebot


Secure Communications
[mysqld]
ssl-ca=ca.pem
ssl-cert=server-cert.pem
ssl-key=server-key.pem

#PerconaLive @RonaldBradford @bytebot


SSL Protocols and Ciphers
mysql> SHOW SESSION STATUS LIKE 'Ssl_version';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| Ssl_version | TLSv1 |
+---------------+-------+
mysql> SHOW SESSION STATUS LIKE 'Ssl_cipher';
+---------------+---------------------------+
| Variable_name | Value |
+---------------+---------------------------+
| Ssl_cipher | DHE-RSA-AES128-GCM-SHA256 |
+---------------+---------------------------+

#PerconaLive @RonaldBradford @bytebot


SSL Client Connections
https://dev.mysql.com/doc/connector-python/en/connector-python-connectargs.html
import mysql.connector
from mysql.connector.constants import ClientFlag

config = {
'user': 'ssluser',
'password': 'asecret',
'host': '127.0.0.1',
'client_flags': [ClientFlag.SSL],
'ssl_ca': '/opt/mysql/ssl/ca.pem',
'ssl_cert': '/opt/mysql/ssl/client-cert.pem',
'ssl_key': '/opt/mysql/ssl/client-key.pem',
}

https://dev.mysql.com/doc/connectors/en/connector-net-tutorials-ssl.html

#PerconaLive @RonaldBradford @bytebot


Secure Connections
● mysql_ssl_rsa_setup in MySQL 5.7
○ This program creates the SSL certificate and key files and RSA key-pair
files required to support secure connections using SSL and secure
password exchange using RSA over unencrypted connections, if those
files are missing.
● use the openssl command

#PerconaLive @RonaldBradford @bytebot


Cloud SSL
● Rackspace - MySQL has GRANT modifier, REQUIRE SSL
○ GRANT SELECT, INSERT, UPDATE, DELETE ON mydatabase.* TO 'user'@'%' REQUIRE
SSL;
● Amazon - ssl-verify-server-cert so SSL connection verifies the DB instance endpoint against the
endpoint in the SSL certificate

mysql \
-h myinstance.123456789012.us-east-1.rds.amazonaws.com \
--ssl-ca=rds-ca-2015-root.pem \
--ssl-verify-server-cert

● Google - max 10 certs/instance

mysql --ssl-ca=server-ca.pem --ssl-cert=client-cert.pem --ssl-key=client-key.pem \


--host=[INSTANCE_IP] --user=root --password

#PerconaLive @RonaldBradford @bytebot


SSL notes
● MySQL clients shipped pre-5.7.3 consider the --ssl option just as advise.
● It silently falls back to unencrypted connections if the server doesn’t accept encrypted
connections!
● You have to configure the instance so that only SSL connections can connect to it.
● Also… Client-side --ssl option is deprecated as of MySQL 5.7.11 and is removed in
MySQL 8.0. For client programs, use --ssl-mode instead. So --ssl-mode=REQUIRED is
preferred
● This matters - your “mysql” client could be from MariaDB Server which is different!
● SSL libraries for MySQL Community and Enterprise are different; also worth noting
some are linked statically, some are dynamic

#PerconaLive @RonaldBradford @bytebot


Authentication Plugins

#PerconaLive @RonaldBradford @bytebot


R
Stronger passwords (sha256)
mysql> CREATE USER bernie@localhost IDENTIFIED WITH sha256_password;
mysql> select host,user,password,plugin,authentication_string,password_expired from mysql.user where
user='bernie'\G
*************************** 1. row ***************************
host: localhost
user: bernie
password:
plugin: sha256_password
authentication_string:
password_expired: N

mysql> SET old_passwords=2;


mysql> SET PASSWORD FOR bernie@localhost = PASSWORD('<clear-text-password>'');
mysql> select host,user,password,plugin,authentication_string,password_expired from mysql.user where
user='bernie'\G
*************************** 1. row ***************************
host: localhost
user: bernie
password:
plugin: sha256_password
authentication_string: $5$ln^h{z\d{JSj}FB$9TF1X0S.Ts4lBSOdsx7p86F4OxRL8ataFok6hyPgew/
password_expired: N

#PerconaLive @RonaldBradford @bytebot


R
PAM
● Percona PAM authentication plugin (5.6+)
○ Open Source, runs community also
● MySQL Enterprise PAM authentication (5.5+)
○ Commercial license
● MariaDB PAM Authentication (5.2+)

https://www.percona.com/doc/percona-pam-for-mysql/index.html
https://dev.mysql.com/doc/refman/5.5/en/pam-pluggable-authentication.html
https://mariadb.com/kb/en/the-mariadb-library/authentication-plugin-pam/

#PerconaLive @RonaldBradford @bytebot


R
Configuring PAM (Demo only)
echo "auth required pam_warn.so
auth required pam_unix.so audit DO NOT
account required pam_unix.so audit" | sudo tee /etc/pam.d/mysqld DO THIS

# DO NOT DO THIS
sudo groupadd shadow
sudo usermod -G shadow mysql
grep shadow /etc/group
sudo chgrp shadow /etc/shadow
sudo chmod 440 /etc/shadow
sudo systemctl restart mysqld.service

#PerconaLive @RonaldBradford @bytebot


R
Installing PAM Plugin
percona56 >SELECT plugin_name FROM information_schema.plugins WHERE plugin_type='AUTHENTICATION';
+-----------------------+
| mysql_native_password |
| mysql_old_password |
| sha256_password |
+-----------------------+

percona56 >INSTALL PLUGIN auth_pam SONAME 'auth_pam.so';


percona56 >SELECT plugin_name FROM information_schema.plugins WHERE plugin_type='AUTHENTICATION';
+-----------------------+
| plugin_name |
+-----------------------+
| mysql_native_password |
| mysql_old_password |
| sha256_password |
| auth_pam |
+-----------------------+
4 rows in set (0.00 sec)

#PerconaLive @RonaldBradford @bytebot


R
Demonstrating PAM
percona56 >CREATE USER external@localhost IDENTIFIED WITH auth_pam;

$ mysql -uexternal
Password:
ERROR 1045 (28000): Access denied for user 'external'@'localhost' (using password: YES)

$ sudo useradd external


$ sudo passwd external

$ mysql -uexternal -p -Ne "SELECT USER()"


+--------------------+
| external@localhost |
+--------------------+

mysql > SELECT /* Use 5.6 */ host, user, password, plugin, authentication_string, password_expired FROM mysql.user WHERE
user='external'\G
*************************** 1. row ***************************
host: localhost
user: external
password:
plugin: auth_pam
authentication_string:
password_expired: N
1 row in set (0.00 sec)

#PerconaLive @RonaldBradford @bytebot


R
MariaDB Authentication Plugins
● edd25519 Authentication (10.1)
Elliptic Curve Digital Signature Algorithm
● GSSAPI Authentication (10.1)
usually synonymous with Kerberos (*NIX)
● Named Pipe (10.1)
Windows
● Unix Socket (10.1)

https://mariadb.com/kb/en/the-mariadb-library/password-authentication-and-encryption-plugins/

#PerconaLive @RonaldBradford @bytebot


R
ed25519 Authentication (MariaDB 10.1+)
maria101 > INSTALL SONAME 'auth_ed25519';
maria101 > CREATE FUNCTION ed25519_password RETURNS STRING SONAME "auth_ed25519.so";

maria101 > SELECT plugin_name FROM information_schema.plugins WHERE plugin_type='AUTHENTICATION';


+-----------------------+
| mysql_native_password |
| mysql_old_password |
| ed25519 |
+-----------------------+

maria101 > SELECT ed25519_password("<clear-text-passwd>") AS pwd;


+---------------------------------------------+
| pwd |
+---------------------------------------------+
| CobnFnV6aei4yy55u90XPFUeBRMBSPtZazzJq8kLHNE |
+---------------------------------------------+

maria101 > CREATE USER demo@localhost IDENTIFIED VIA ed25519 USING 'CobnFnV6aei4yy55u90XPFUeBRMBSPtZazzJq8kLHNE';

#PerconaLive @RonaldBradford @bytebot


R
Unix Socket Authentication (MariaDB 10.1)
maria101 > INSTALL PLUGIN unix_socket SONAME 'auth_socket';
maria101 > SELECT plugin_name FROM information_schema.plugins WHERE plugin_type='AUTHENTICATION';
+-----------------------+
| mysql_native_password |
| mysql_old_password |
| ed25519 |
| unix_socket |
+-----------------------+

maria101 > CREATE USER vagrant IDENTIFIED VIA unix_socket;

$ whoami
vagrant

$ mysql -e "SELECT USER(),CURRENT_USER()"


+-------------------+----------------+
| USER() | CURRENT_USER() |
+-------------------+----------------+
| vagrant@localhost | @localhost |
+-------------------+----------------+

root@maria101 > CREATE USER notvagrant IDENTIFIED VIA unix_socket;

$ mysql -unotvagrant
ERROR 1698 (28000): Access denied for user 'notvagrant'@'localhost'

#PerconaLive @RonaldBradford @bytebot


R
Auditing

#PerconaLive @RonaldBradford @bytebot


R
Auditing Capabilities
MySQL

- Logging account access

- Logging SQL statements

- Logging uncommon SQL patterns

OS

- Logging account logins

- Logging sudo commands


#PerconaLive @RonaldBradford @bytebot
R
Auditing Plugin Options
● McAfee (5.1+)
● Percona (5.5)
● MariaDB (10.0)
● MySQL Enterprise (5.5)

● Installation
● Configuration
● Log format

#PerconaLive @RonaldBradford @bytebot


R
Installing McAfee Plugin (on Percona 5.6)
wget
https://dl.bintray.com/mcafee/mysql-audit-plugin/:audit-plugin-mysql-5.6-1.1.5-742-linux-x86_64.z
ip
unzip audit-plugin-mysql-5.6-1.1.5-742-linux-x86_64.zip
cd audit-plugin-mysql-5.6-1.1.5-742/
sudo cp lib/libaudit_plugin.so /usr/lib64/mysql/plugin/
sudo chmod +w /usr/lib64/mysql/plugin/libaudit_plugin.so
wget https://raw.github.com/mcafee/mysql-audit/master/offset-extract/offset-extract.sh
chmod +x offset-extract.sh
sudo yum install -y gdb
./offset-extract.sh `which mysqld`
sudo vi /etc/my.cnf

plugin-load=AUDIT=libaudit_plugin.so
audit_offsets=6992, 7040, 4000, 4520, 72, 2704, 96, 0, 32, 104, 136, 7128, 4392, 2800, 2808,
2812, 536, 0, 0, 6360, 6384, 6368, 13048, 548, 516

sudo service mysqld restart


https://github.com/mcafee/mysql-audit/wiki/Installation

#PerconaLive @RonaldBradford @bytebot


R
Installing Audit plugin (Enterprise 5.7)
enterprise57 > SHOW GLOBAL VARIABLES LIKE 'audit%';
Empty set (0.01 sec)

enterprise57 $ mysql -uroot -p < /usr/share/mysql/audit_log_filter_linux_install.sql


Result
OK

enterprise57 > SELECT PLUGIN_NAME, PLUGIN_STATUS


FROM INFORMATION_SCHEMA.PLUGINS
WHERE PLUGIN_NAME LIKE 'audit%';
+-------------+---------------+
| PLUGIN_NAME | PLUGIN_STATUS |
+-------------+---------------+
| audit_log | ACTIVE |
+-------------+---------------+
1 row in set (0.00 sec)

https://dev.mysql.com/doc/refman/5.5/en/audit-log.html

#PerconaLive @RonaldBradford @bytebot


R
Installing Percona Audit Plugin
percona56 $ mysql -uroot

percona56 >INSTALL PLUGIN audit_log SONAME 'audit_log.so';

percona56 >SELECT PLUGIN_NAME, PLUGIN_STATUS FROM INFORMATION_SCHEMA.PLUGINS WHERE PLUGIN_NAME


LIKE 'audit%';
+-------------+---------------+
| PLUGIN_NAME | PLUGIN_STATUS |
+-------------+---------------+
| audit_log | ACTIVE |
+-------------+---------------+

https://www.percona.com/doc/percona-server/5.5/management/audit_log_plugin.html
https://www.percona.com/blog/2014/05/16/introduction-to-the-percona-mysql-audit-log-plugin/

#PerconaLive @RonaldBradford @bytebot


R
Installing MariaDB Audit Plugin
maria101 > INSTALL PLUGIN server_audit SONAME 'server_audit.so';

maria101 > SELECT PLUGIN_NAME, PLUGIN_STATUS FROM INFORMATION_SCHEMA.PLUGINS WHERE


PLUGIN_TYPE='AUDIT';
+--------------+---------------+
| PLUGIN_NAME | PLUGIN_STATUS |
+--------------+---------------+
| SERVER_AUDIT | ACTIVE |
+--------------+---------------+

maria101 > SET GLOBAL server_audit_events = 'CONNECT,QUERY,TABLE';


maria101 > SET GLOBAL server_audit_logging=ON;

https://mariadb.com/kb/en/mariadb/about-the-mariadb-audit-plugin/

#PerconaLive @RonaldBradford @bytebot


R
Audit Log Formats
● OLD (XML)
● NEW (XML)
● JSON
● CSV

https://dev.mysql.com/doc/refman/5.5/en/audit-log-file.html

#PerconaLive @RonaldBradford @bytebot


R
OLD/NEW Format
$ sudo tail /var/lib/mysql/audit.log $ sudo tail /var/lib/mysql/audit.log
<AUDIT_RECORD <AUDIT_RECORD>
NAME="Connect" <NAME>Quit</NAME>
RECORD="9_2017-09-04T21:40:19" <RECORD>1358_2017-09-25T12:59:46</RECORD>
TIMESTAMP="2017-09-04T21:45:34 UTC" <TIMESTAMP>2017-09-25T12:59:49
CONNECTION_ID="13" UTC</TIMESTAMP>
STATUS="1045" <CONNECTION_ID>1</CONNECTION_ID>
USER="root" <STATUS>0</STATUS>
PRIV_USER="root" <USER>root</USER>
OS_LOGIN="" <PRIV_USER>root</PRIV_USER>
PROXY_USER="" <OS_LOGIN></OS_LOGIN>
HOST="localhost" <PROXY_USER></PROXY_USER>
IP="" <HOST>localhost</HOST>
DB="" <IP></IP>
/> <DB></DB>
</AUDIT_RECORD>

#PerconaLive @RonaldBradford @bytebot


R
JSON/CSV Format
sudo tail /var/lib/mysql/audit.log

{"audit_record":{"name":"Ping","record":"3417_2017-09-25T13:37:27","timestamp":"2017-09-25T13:37:
29 UTC","command_class":"error","connection_id":"1","status":0,"sqltext":"","user":"root[root] @
localhost []","host":"localhost","os_user":"","ip":"","db":""}}
{"audit_record":{"name":"Quit","record":"3418_2017-09-25T13:37:27","timestamp":"2017-09-25T13:37:
29
UTC","connection_id":"1","status":0,"user":"root","priv_user":"root","os_login":"","proxy_user":"
","host":"localhost","ip":"","db":""}}

sudo tail /var/lib/mysql/audit.log

"Connect","4915_2017-09-25T13:39:34","2017-09-25T13:39:38
UTC","1",0,"root","root","","","localhost","",""
"Ping","4916_2017-09-25T13:39:34","2017-09-25T13:39:38 UTC","error","1",0,"","root[root] @
localhost []","localhost","","",""

#PerconaLive @RonaldBradford @bytebot


R
McAfee Audit Log Example
mysql56 >set global audit_json_file=ON;

$ mysql -udemo -px


Warning: Using a password on the command line interface can be insecure.
ERROR 1045 (28000): Access denied for user 'demo'@'localhost' (using password: YES)

$ sudo cat /var/lib/mysql/mysql-audit.json


{"msg-type":"activity","date":"1505267816852","thread-id":"13","query-id":"0","user":"demo","priv
_user":"demo","ip":"","host":"localhost","connect_attrs":{"_os":"Linux","_client_name":"libmysql"
,"_pid":"5944","_client_version":"5.6.37","_platform":"x86_64","program_name":"mysql"},"pid":"594
4","os_user":"vagrant","appname":"mysql","status":"1045","cmd":"Failed Login","query":"Failed
Login"}
{"msg-type":"activity","date":"1505267816852","thread-id":"13","query-id":"0","user":"demo","priv
_user":"demo","ip":"","host":"localhost","connect_attrs":{"_os":"Linux","_client_name":"libmysql"
,"_pid":"5944","_client_version":"5.6.37","_platform":"x86_64","program_name":"mysql"},"pid":"594
4","os_user":"vagrant","appname":"mysql","cmd":"Connect","query":"Connect"}

#PerconaLive @RonaldBradford @bytebot


R
McAfee Audit Log Example
{
"msg-type": "activity",
"date": "1505267816852",
"thread-id": "13",
"query-id": "0",
"user": "demo",
"priv_user": "demo",
...
"pid": "5944",
"os_user": "vagrant",
"appname": "mysql",
"status": "1045",
"cmd": "Failed Login",
"query": "Failed Login"
}

#PerconaLive @RonaldBradford @bytebot


R
MariaDB Audit Log Format
$ sudo cat /var/lib/mysql/server_audit.log
20170912 02:02:15,maria101,root,localhost,3,11,QUERY,,'SET GLOBAL server_audit_logging=ON',0
20170912 02:02:39,maria101,root,localhost,3,0,DISCONNECT,,,0
20170912 02:02:40,maria101,root,localhost,4,0,CONNECT,,,0
20170912 02:02:40,maria101,root,localhost,4,13,QUERY,,'select @@version_comment limit 1',0
20170912 02:02:40,maria101,root,localhost,4,14,QUERY,,'select USER()',0
20170912 02:02:42,maria101,root,localhost,4,0,DISCONNECT,,,0
20170912 02:02:45,maria101,root,localhost,5,0,FAILED_CONNECT,,,1045
20170912 02:02:45,maria101,root,localhost,5,0,DISCONNECT,,,0

https://mariadb.com/kb/en/library/mariadb-audit-plugin-log-format/

#PerconaLive @RonaldBradford @bytebot


R
Auditing Implementation
● MariaDB Server
○ User filtering as an additional feature via audit API extensions
○ Query cache enabled? No table records
● Percona
○ Multiple output formats: OLD, NEW, JSON, CSV
○ Filter by user, SQL command type, database,
○ Auditing can be expensive, so asynchronous/performance/semisynchronous/synchronous
modes for logging - e.g. log using memory buffers, drop messages if buffers are full, or log
directly to file, flush and sync at events, etc.
● McAfee Audit plugin
○ Uses offsets
● MySQL Enterprise Audit Plugin (utility: mysqlauditgrep)

#PerconaLive @RonaldBradford @bytebot


R
Encryption

#PerconaLive @RonaldBradford @bytebot


Secure Storage
● Encryption of data at rest
○ Data (table vs tablespace)
○ Binary Logs
○ Other Logs

● Key management

#PerconaLive @RonaldBradford @bytebot


Encryption in MariaDB Server
● Encryption: tablespace OR table level encryption with support for rolling keys
using the AES algorithm (only with a keyserver)
○ table encryption — PAGE_ENCRYPTION=1
○ tablespace encryption — encrypts everything including log files
● file_key_management_filename, file_key_management_filekey,
file_key_management_encryption_algorithm
● Documented — https://mariadb.com/kb/en/mariadb/data-at-rest-encryption/
● Tablespace/logs scrubbing: background process that regularly scans through
the tables and upgrades the encryption keys
● --encrypt-tmp-files & --encrypt-binlog

#PerconaLive @RonaldBradford @bytebot


Encryption in MariaDB Server II
[mysqld] CREATE TABLE customer (
plugin-load-add=file_key_management.so customer_id bigint not null primary
file-key-management key,
file-key-management-filename = customer_name varchar(80),
/home/mdb/keys.enc
customer_creditcard varchar(20))
innodb-encrypt-tables
ENGINE=InnoDB
innodb-encrypt-log
innodb-encryption-threads=4 page_encryption=1
aria-encrypt-tables=1 # PAGE row format page_encryption_key=1;
encrypt-tmp-disk-tables=1 # this is for Aria

#PerconaLive @RonaldBradford @bytebot


Encryption in MariaDB Server III
● Use the preset! - /etc/my.cnf.d/enable_encryption.preset
● A plugin for Amazon Key Management Server (KMS)
● mysqlbinlog has no way to read (i.e. decrypt) an encrypted binlog
● This does not work with MariaDB Galera Cluster yet (gcache is not encrypted
yet), and also Percona XtraBackup needs additional work (i.e. if you encrypt
the redo log)
○ Recommended backup method: MariaDB Backup (fork of Percona XtraBackup)

#PerconaLive @RonaldBradford @bytebot


Encryption in MySQL
● MySQL 5.7.11 introduces InnoDB tablespace encryption
● early-plugin-load=keyring_file.so in my.cnf
● Must use innodb_file_per_table
● Convert via ALTER TABLE table ENCRYPTION=‘Y’
● Data is not encrypted in the redo/undo/binary logs
● Has external key management (Oracle Key Vault)

#PerconaLive @RonaldBradford @bytebot


Key management in MySQL
● MySQL 5.7.11 and higher includes a keyring plugin, keyring_file, that stores
keyring data in a file local to the server host.
● MySQL 5.7.12 and higher includes keyring_okv, a KMIP 1.1 plugin for use
with KMIP-compatible back end keyring storage products such as Oracle Key
Vault and Gemalto SafeNet KeySecure Appliance. (Enterprise)
● MySQL 5.7.19 and higher includes keyring_aws, a plugin that communicates
with the Amazon Web Services Key Management Service for key generation
and uses a local file for key storage. (Enterprise)
● MySQL 5.7.13 and higher includes an SQL interface for keyring key
management, implemented as a set of user-defined functions (UDFs).

#PerconaLive @RonaldBradford @bytebot


Encryption conclusions
● InnoDB Tablespace Encryption: MySQL & Percona Server 5.7
○ MySQL Enterprise 5.7: Transparent Data Encryption
■ MySQL Enterprise TDE uses a two-tier encryption key architecture, consisting of a master encryption key and
tablespace keys, which provides easy key management and rotation.
● Google patches for InnoDB/XtraDB/Aria (tmptable) tablespace encryption: MariaDB Server 10.1+
● Eperi patch for InnoDB/XtraDB table encryption: MariaDB Server 10.1+
● AWS KMS: MariaDB Server 10.2+ (in 10.1 you have to compile it)
● Linux Unified Key Setup (LUKS)
● Column encryption doesn’t exist today
● Other solutions include Penta Security’s MyDiamo
● Cloud? Amazon has its own encryption + KMS + IAM. Google uses AES-256, with symmetric keys.
Rackspace has a MariaDB Server encryption option.

#PerconaLive @RonaldBradford @bytebot


Linux Unified Key Setup (LUKS)
Using crypt+LUKS, we can encrypt everything (data + logs) under one umbrella –
provided that all files reside on the same disk. If you separate the various logs on
to different partitions, you will have to repeat for each partition.

● Creating, Formatting and Mounting an Encrypted Disk


● Creating a Backup of Encryption Information
● Unmounting and Closing a Disk
● Rotating Keys (Adding / Removing Keys)

https://www.percona.com/blog/2017/06/06/mysql-encryption-at-rest-part-1-luks/

#PerconaLive @RonaldBradford @bytebot


SQL Standard Roles
● Bundles users together, with similar privileges - follows the SQL standard
● MariaDB Server 10.0 (10.1 adds that each user can have a DEFAULT ROLE)
● MySQL 8.0 DMR

CREATE ROLE audit_bean_counters;


GRANT SELECT ON accounts.* to audit_bean_counters;
GRANT audit_bean_counters to ceo;

https://mariadb.com/kb/en/mariadb/roles_overview/
https://dev.mysql.com/doc/refman/8.0/en/roles.html

#PerconaLive @RonaldBradford @bytebot


Firewalls

#PerconaLive @RonaldBradford @bytebot


Firewall Implementation
MariaDB - MariaDB MaxScale Firewall Filter

● https://mariadb.com/resources/blog/maxscale-firewall-filter

ProxySQL Firewall http://www.proxysql.com/

MySQL - MySQL Enterprise Firewall

● https://www.mysql.com/products/enterprise/firewall.html

Datasunrise firewall for Percona

● https://www.datasunrise.com/firewall/percona/
#PerconaLive @RonaldBradford @bytebot
MariaDB MaxScale
● Pre-2.0, GPLv2; post-2.0 BSL license (also, performance improvements;
SQLite parser)
● The database firewall filter is used to block queries that match a set of rules. It
can be used to prevent harmful queries from reaching the backend database
instances or to limit access to the database based on a more flexible set of
rules compared to the traditional GRANT-based privilege system. Currently
the filter does not support multi-statements.
● Needs configuration via whitelist/blacklist
● Data masking - possible to obfuscate the returned value of a particular
column (e.g. don’t allow SSN numbers to go back)

#PerconaLive @RonaldBradford @bytebot


AirBnB MaxScale fork
“The query blocklisting feature leverages the MySQL parse tree to look for
existence of malformed predicate conditions in update and delete statements and
reject such statements. For more protective coverage, we block MySQL update
and delete statements without any predicate condition as well. We deployed
Airbnb MaxScale with query blocklist feature and it has protected us from at least
one instance of scary corrupted query that could have caused damage to one of
our core database tables.”

https://medium.com/airbnb-engineering/unlocking-horizontal-scalability-in-our-web
-serving-tier-d907449cdbcf

#PerconaLive @RonaldBradford @bytebot


ProxySQL
● Has firewall and data masking
● Query rules based on Google’s RE2-style regex
● Query firewall - queries don’t have to hit the database
● Validate query changes without impacting production database (“mirror”)
○ Match? Thread is cloned, inherits all attributes of original thread (can also redirect to a
different hostgroup)
○ Upgrades? E.g. in 5.7 GENERATED is a reserved word; you can get it to rewrite queries
● Data masking to obfuscate the returned value of a particular column (e.g.
don’t allow SSN numbers to go back)

#PerconaLive @RonaldBradford @bytebot


MySQL Enterprise Firewall
● Arrived in MySQL 5.6
● Block SQL injection attacks that don’t match the whitelist
● Intrusion detection to detect what’s not matching a whitelist and alert DBA
● Block out of policy SQL statements
● Enforce the whitelist
● Record approved incoming SQL statements; build whitelists on a per user
basis
● Log SQL activity

#PerconaLive @RonaldBradford @bytebot


Firewall conclusions
● Your only real choice is ProxySQL - GPLv3 friendly licensing
● MySQL Enterprise Firewall will cost you money
● MariaDB MaxScale is BSL licensed (which is versioned, currently 1.1)
○ Additional Use Grant: You may use the Licensed Work when your application uses the
Licensed Work with a total of less than three server instances for any purpose.
○ The Licensor hereby grants you the right to copy, modify, create derivative works, redistribute,
and make non-production use of the Licensed Work. The Licensor may make an Additional
Use Grant, above, permitting limited production use.
○ https://github.com/mariadb-corporation/MaxScale/blob/2.1/LICENSE.TXT

#PerconaLive @RonaldBradford @bytebot


SQL Injection
● Always using bind variables
● Escape input content
● Restricted "least" privileges
○ Do not have GRANT ALL

#PerconaLive @RonaldBradford @bytebot


Other Security Considerations

#PerconaLive @RonaldBradford @bytebot


Improving Database Server Access
● Restricting user access to your database server (login accounts)
○ Every physical person
■ dedicated OS login
■ dedicated database login
○ sudo restrictions (e.g. sudo su -)
■ Setup sudo group
■ Grant only specific commands to execute
○ Never share account details
○ Password expiry

● MFA

#PerconaLive @RonaldBradford @bytebot


Improving Database Server Access
● Restricting traffic to your database server (open ports)
● Run a software firewall
○ iptables, ufw

● You should use OS software meant to benefit security


○ SELinux / Apparmor

#PerconaLive @RonaldBradford @bytebot


Improving Data Access
● Restrict access to datadir
○ Don't put error log here
○ Don't put socket file here
○ Don't put pid file here

● Restrict access to view mysql.user table on filesystem


○ Check out the examples of how to Hack MySQL

http://effectivemysql.com/downloads/MySQLSecurityEssentialsPerconaLive2015.pdf

#PerconaLive @RonaldBradford @bytebot


Installation
● Using your Linux distribution… mostly gets you MariaDB when you ask for
mysql-server
○ Except on Debian/Ubuntu
■ However, when you get mariadb-server, you get an authentication plugin — auth_socket
for “automatic logins”
○ You are asked by debhelper to enter a password
● You can use the APT/YUM repositories from Oracle MySQL, Percona or
MariaDB
● Don’t disable SELinux: system_u:system_r:mysqld_t:s0

#PerconaLive @RonaldBradford @bytebot


Update Cadence
A security patch is so named because it improves security and generally
addresses a means of attack of your system

● OS
● Database
● Application Stack

Why are you still running MySQL 5.5 or older?

#PerconaLive @RonaldBradford @bytebot


Deployment Security
Who has control over running deployments?

i.e. deploying code that manipulates your data or structure

An application user SHOULD NOT have CREATE, ALTER, DROP privileges

● User to write data


● User to read data
● DBA to administer data (restricted to localhost)

#PerconaLive @RonaldBradford @bytebot


Use of Docker Containers
Docker shows a disregard for security with 'root' OS logins by default

● MySQL server installation approach via exposed passwords


● Configuration is contained with container
○ Can you access the container via SSH
○ Can you copy and use container

#PerconaLive @RonaldBradford @bytebot


Reference Material
https://www.mysql.com/why-mysql/presentations/mysql-security-best-practices/

● MySQL Authentication and Password Policies


● MySQL Authorization and Privilege Management
● MySQL Encryption to secure sensitive data
● MySQL Enterprise Firewall to block database attacks such as an SQL
Injection
● MySQL Enterprise Audit to implement policy

#PerconaLive @RonaldBradford @bytebot


MySQL Manual Security Practices
http://dev.mysql.com/doc/refman/5.5/en/security.html

http://dev.mysql.com/doc/refman/5.6/en/security.html

http://dev.mysql.com/doc/refman/5.7/en/security.html

#PerconaLive @RonaldBradford @bytebot


(some of) Your questions answered :-)
● Can you have SSL config options in mysql_config_editor?
○ Yes! Use login-paths.
http://mysqlhighavailability.com/new-mysql-utilities-now-supports-ssl-and-configuration-files/
● In 8.0, you want to define mandatory roles with SET PERSIST?
○ Read: https://dev.mysql.com/doc/refman/8.0/en/roles.html which saves the value for
subsequent server restarts
● Using Amazon AWS RDS and need the audit plugin via an option group?
○ Read:
http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.MySQL.Options.AuditPl
ugin.html (this works with MariaDB Server and MySQL)

#PerconaLive @RonaldBradford @bytebot


Thank You Sponsors!

12
1
SAVE THE DATE! April 23-25, 2018
Santa Clara Convention Center

12
CALL FOR PAPERS OPENING SOON!
2 www.perconalive.com
Rate us! Thanks/Q&A

● Don’t forget to rate us in the app!


● Ronald Bradford: @RonaldBradford /
http://ronaldbradford.com/blog
● Colin Charles: @bytebot /
http://www.bytebot.net/blog/

#PerconaLive @RonaldBradford @bytebot

You might also like