A Checklist for Preventing

Common But Deadly

MySQL® Problems

Part 2

A Percona White Paper

October 22, 2014
 Contents

Introduction ................................................................................................................ 3
The Busy MySQL DBA .............................................................................................. 3
The Impact on Your Business .................................................................................. 3
The Common but Deadly MySQL Problems ................................................................ 4
Poor Migration Preparation .................................................................................... 4
Upgrades ............................................................................................................. 4
Capacity Alterations ............................................................................................ 5
Moving Operations Into or Out of the Cloud ...................................................... 5
Changing Products .............................................................................................. 6
Poor Query Performance ........................................................................................ 7
Lack of High Availability/Too Much Downtime ....................................................... 8
How Percona Can Help.............................................................................................. 10
About Percona .......................................................................................................... 11

December 12, 2014 2

©Percona. All rights reserved.
 Introduction
This is the second in a series of three white papers for busy DBAs and their
management. The series provides a checklist of some of the most common MySQL
implementation, performance, and recovery problems that can be minimized or
prevented by proactive MySQL administration best practices. The issues covered in this
white paper are:

● Poor Migration Preparation

● Poor Query Performance
● Lack of High Availability/Too Much Downtime

The first white paper in this series covers Out of Date MySQL Versions, Inadequate
MySQL Configuration Settings, and Unmindful Deployment Practices and can be
downloaded from the Percona website.

The information in this series of white papers is based on collected advice from the
Percona Support, Consulting, and Managed Services teams. These teams include experts
in MySQL operations who have helped thousands of clients over the past eight years.

The Busy MySQL DBA

Busy MySQL DBAs often fall into one of three categories:

 You may be filling the role of your organization’s MySQL DBA because there is
no one else available. You may have limited DBA experience and might be an IT
professional, a SysAdmin, the CTO, or on the Engineering staff.
 You may be a DBA by profession but be overloaded due to lack of resources in
your organization. Your time might be divided between database operations
and other tasks such as application development. You may be addressing issues
reactively or when time allows, hopefully before disaster strikes.
 You may be a DBA transitioning into a more strategic role in which the handling
of common operational tasks is no longer your main focus or desire. In this
capacity, common but necessary database administrative tasks are performed
ad hoc, as time allows, or perhaps not at all.

The Impact on Your Business

As a busy DBA or the manager of a busy DBA, you must consider the problems caused by
part-time or reactive MySQL administration and the impact those issues may have on
your business. It is management’s responsibility to maintain the resources necessary to
manage your organization’s MySQL environments. A lack of resources can lead to:

 Unacceptable downtime
 Unnecessary costs

December 12, 2014 3

©Percona. All rights reserved.
  Potential malicious attacks
 Frustrated users
 Lost business

Percona can help. Percona Consulting and Support services for MySQL offer customized
help and immediate assistance from MySQL experts. Percona Managed Services for
MySQL can take over your organization’s MySQL operational tasks, letting you
concentrate on other projects.

The Common but Deadly MySQL Problems

We will now take a look at some common MySQL implementation, performance,
and recovery problems.

Poor Migration Preparation

There may come a time in your application’s lifecycle when it makes sense to
migrate to a new infrastructure. This may involve moving from:

 One variant of MySQL to another

 Single node MySQL to a cluster environment
 On premise MySQL to the cloud
 Your current infrastructure to different hardware

A migration should result in a positive impact on your bottom line. However,

migrations face a number of potential issues that can bring down your system, and
your business, for an extended period of time. Upgrades, capacity alterations,
moving operations in or out of the cloud, and changing products can result in
downtime if not properly planned.

Upgrades
Upgrading an operating system or MySQL version might require a primary master
to be taken offline. An upgrade can be facilitated by promoting a slave to take over
the primary master’s role for the duration of the upgrade. However, this method
requires careful planning and knowledge of potential problems. In some cases,
erroneous writes can be made to the demoted master resulting in data integrity
issues. Replication reconfiguration mistakes can result in missing transactions or
data consistency issues. Both of these potential pitfalls cost organizations extra
time and effort to manually solve the problems introduced in the upgrade.

When preparing for an upgrade in this manner, there are several things you can do
to minimize disruption. First, make sure that you isolate the node from potential
connection sources. You will need to work with the team responsible for the
application to ensure that threads are not being created on the host that is

December 12, 2014 4

©Percona. All rights reserved.
 undergoing the maintenance. Percona uses virtual IP addresses to relocate roles
between nodes and then we use MHA to perform the reconfiguration of slaves.

The integrity of the data on the slave must be trusted when moving the read/write
role to an asynchronous slave because there is no guarantee from MySQL that all
events have been replayed on the slave. The best way to achieve this is to use the
Percona Toolkit tool pt-table-checksum regularly and action the sync of any
detected differences with pt-table-sync. It is a best practice to make a backup of
the data before executing the upgrade so you have a rollback plan. To further
reduce the margin for error, we recommend that a detailed action plan is produced
and peer reviewed to make sure nothing is missed.

Capacity Alterations
You may need to take a server offline to perform hardware maintenance such as
fitting memory. There are also cases where your primary master may be on
hardware over-specified for its workload and you want to reduce its capacity.
Minimizing downtime through the use of High Availability tools can reduce or even
remove the need to take you application offline while work is performed.

For Percona, Master High Availability (MHA) with MySQL replication allows us to
automate the practice of relocating a Virtual IP address and reconfiguring slave
replication coordinates from manual atomic individual tasks to a smart and swift
process. The technique has permitted us to grow block storage by terabytes
without end users noticing any service interruptions. We recently diagnosed that a
client had over specified their hardware and were paying for expensive high
performance storage capable of many times the required throughput. Through the
user of Percona’s HA toolset, we were able to clone the master and migrate
without suffering any outages. In the end, we significantly reduced our customer’s
total cost for their overall system.

Moving Operations Into or Out of the Cloud

Moving from one cloud provider to another or even between services of the same
cloud provider (e.g., AWS RDS and EC2) also requires careful planning. For example,
the route from RDS MySQL version 5.5 to EC2 is not an obvious one. Many
organizations will justify losing business to make the change. Other organizations
may decide that the migration is not justified and remain on legacy versions which
lack new functionality or performance improvements. Switchover must be expertly
timed and handled with minimal interruption so operations can continue
unaffected and users do not experience any availability issues.

Percona recently managed a multi-terabyte migration from legacy AWS to Amazon

Virtual Private Cloud that provided a 24x7 online gaming client bigger, newer

December 12, 2014 5

©Percona. All rights reserved.
 hardware. Our Managed Services team, which handles this type of operation
regularly, completed the migration without any interruption of service. We were
able to plan and execute this migration by using Percona XtraBackup as a means of
cloning the nodes and MHA to provide replication failover to our preferred new
node while completing the task of reconfiguring all slaves for the new source. This
protocol permitted operations to be performed completely online.

Changing Products
Some organizations migrate from one variant of MySQL to another to make use of
functionality that does not exist in their current variant. Taking time to plan and
identify the issues which may result in downtime can highlight where
measurements need to be made to minimize disruptions to daily workflow.

If you take the database offline while making changes, operations may be
interrupted. Percona has developed techniques to minimize or eradicate downtime
over many years of implementing migrations for clients. In many cases, Percona
uses the MHA software package to simply and safely move the role of master to a
waiting slave host. This provides the agility to failover to a machine that has had
upgrades applied or that has been built in the new datacenter.

Related Percona Resources

There are several Percona resources on the Percona website that you can use to
help better prepare for a migration including:

 Webinar: Upgrading to MySQL 5.6: Best Practices given by Peter Zaitsev,

Percona CEO and Founder
 Webinar: Migrating to Percona XtraDB Cluster given by Percona Managing
Consultant Jay Janssen
 Webinar: Running MySQL 5.6 on Amazon RDS given by Percona Consulting
 Percona’s MySQL Performance Blog, including these posts:
 MySQL Upgrade Best Practices, by Percona Software Engineer Muhammad
Irfan
 What I learned while migrating a customer MySQL installation to Amazon
RDS, by Percona Managing Consultant Michael Coburn
 Multi-Valued INSERTs, AUTO_INCREMENT & Percona XtraDB Cluster, by
Percona Sr. Architect Ernie Souhrada
 Percona Open Source Software Manual Repository. Software manuals include
upgrade guidance for Percona software including Percona Server and Percona
XtraDB Cluster.
 Percona Community Forum: MySQL and Percona Server Sub-Forum
 Percona Community Forum: Percona XtraDB Cluster Sub-Forum

December 12, 2014 6

©Percona. All rights reserved.
 Poor Query Performance
Query tuning is one of the most important ways you can improve the performance
of your MySQL server, yet it is often overlooked. Bad queries can overload your
server or affect user response time. Users do not tolerate poor response time. If a
user has to wait, they may leave. If they do not leave, they may retry their previous
action which further increases the number of concurrent bad queries running on
your server. Too many poorly performing queries may result in hitting the
max_connections on your server which will become overloaded and unresponsive.

A good index on a simple select can be the difference between response times
measured in minutes or milliseconds. For most of our clients experiencing excessive
server loads or slow MySQL, the root cause is a poor performing query or queries
which can be easily fixed with the correct indexes or by rewriting the query.

We have improved response times, reduced server load and improved scalability
with our query optimization services. This allows our clients to serve more
concurrent users without affecting performance.

The first step before making any changes to your application is measuring your
query performance. Percona uses tools like pt-query-digest to identify the worst
performing queries and quantify the results of a change. We also use Percona
Cloud Tools to monitor how the worst performing queries might change
throughout the day and especially when new code is deployed.

Related Percona Resources

There are several Percona resources on www.percona.com that can help improve
your query performance including:

 eBook: MySQL Query Optimization, by Percona CEO Peter Zaitsev, CTO Vadim
Tkachenko, and Software Engineer Muhammad Irfan
 eBook: MySQL Logs, by Percona CEO Peter Zaitsev, Principal Architect Peter
Boros, Sr Knowledge Manager Bill Karwin, and Software Engineer Aleksandr
Kuzminsky
 Webinar: Advanced Query Tuning in MySQL 5.6 and Beyond given by Percona
Principal Architect Alexander Rubin
 Webinar: Practical MySQL Performance Optimization given by Percona CEO
Peter Zaitsev
 Webinar: MySQL Query Patterns, Optimized given by Sr Knowledge Manager,
Percona Support Team, Bill Karwin
 Webinar: MySQL 5.6 Indexing: Best Practices given by Percona CEO Peter
Zaitsev

December 12, 2014 7

©Percona. All rights reserved.
  Webinar: Analyze MySQL Query Performance with Percona Cloud Tools given
by Percona CTO Vadim Tkachenko
 Percona Cloud Tools is a hosted service providing access to query performance
insights for all MySQL uses. Sign up for free.
 Percona’s MySQL Performance Blog, including these posts:
 How MySQL ‘queries’ and ‘questions’ are measured, by Percona CEO Peter
Zaitsev
 Tools and tips for analysis of MySQL’s Slow Query Log, by Percona Software
Engineer Muhammed Irfan

Lack of High Availability/Too Much Downtime

A MySQL architecture with a single node has the potential to significantly impact
application availability. But even a multi-node MySQL architecture with a master
and slaves using standard asynchronous replication has risk if failover is manual.
Manually failing over can take time and requires precision that can be lost when
performed during a disaster. Downtime of production databases typically results in
a major impact on service quality, a poor user experience, and potential lost
business.

In order to minimize downtime, organizations can pursue a high availability (HA)

strategy which incorporates automation. Automating failover processes potentially
reduces failover time from minutes to seconds and minimizes human error.

High availability consists of two core needs:

 Data redundancy which can be accomplished with native MySQL

replication, solutions such as Percona XtraDB Cluster (which
incorporates Galera technology), lower level solutions such as
DRBD, and other storage-based software and technology.

 Automated failover which can use solutions based on virtual IPs,

AWS elastic IPs, or load balancing solutions such as HAProxy

While it is imperative to minimize downtime, you may lack the resources to

implement a proven high availability strategy. It is best to avoid an over-
engineered solution. It is preferable to opt for a solution you can easily
troubleshoot if required. Your chosen solution should be tried and tested so you
can be confident it will work when you need it. Among other characteristics, your
strategy should include:

December 12, 2014 8

©Percona. All rights reserved.
  A warmed cache on a standby host to reduce the time to recover
from a failure

 Suitable backup hosts to cope with the load within acceptable

tolerances after a failover

There are many high availability options available. The right solution depends on
your application’s unique needs. For example, a Percona client in the retail industry
had implemented Percona XtraDB Cluster with HAProxy and came to Percona for
help with poor availability. We analyzed their environment and found that their
approach was not a match for the workload of their application. In their case, large
transactions were affecting the availability of the cluster. We helped migrate them
to a solution based on native MySQL replication, MHA, and AWS EIP failover. The
result was an improvement in availability and a simple failover that suited their
application’s workload.

Finding a high availability solution that is right for your business requires in-depth
knowledge of the options. We recommend working with an experienced team that
understands the choices, how to assess which is suitable for your situation, and can
provide guidance on implementation matters.

Related Percona Resources

There are several Percona resources on www.percona.com that you can use to
increase availability and reduce downtime:

 Webinar: 7 Key MySQL Clustering Technologies to Consider: A Joint 451

Research/Percona Webinar given by Percona CEO Peter Zaitsev and 451
Research’s Matt Aslett, Research Director of Data Management and Analytics
 Webinar: Choosing a MySQL High Availability Solution given by Senior
Architect Ernie Souhrada
 Webinar: MySQL High Availability and Geographical Disaster Recovery with
Percona Replication Manager given by Percona Principal Architect Yves
Trudeau
 Webinar: Using MySQL Global Transaction IDs in Production, by Percona
Architect Stephane Combaudon
 White Paper: High Level Multi-Datacenter MySQL High Availability, by Percona
Managing Consultant Jay Janssen
 Video: OpenStack: A DBA Perspective given by Percona Principal Architect
Alexander Rubin

December 12, 2014 9

©Percona. All rights reserved.
  Video: 5 Proactive Measures to Reduce Downtime, given by Percona Sr.
Consultant Fernando Ipar
 eBook: MySQL Fabric Part 1, by Percona Sr. Consultant Fernando Ipar and
Remote DBA Martin Arrieta
 Case Study: PagerDuty Relies on Percona for Its MySQL Cluster in the Cloud
 Percona’s MySQL Performance Blog, including these posts:
 High Availability with MySQL Fabric, by Percona Sr. Consultant Fernando Ipar
and Remote DBA Martin Arrieta
 Prevent MySQL downtime: Set max_user_connections, by Percona CEO Peter
Zaitsev
 Failover with the MySQL Utilities: Part 1 (mysqlrpladmin) and Part 2
(mysqlfailover), by Percona Architect Stephane Combaudon
 The use of Iptables ClusterIP target as a load balancer for PXC, PRM, MHA and
NDB, by Percona Principal Architect Yves Trudeau
 HAProxy: Give me some logs on CentOS 6.5!, by Percona Architect Stephane
Combaudon

How Percona Can Help

Managing your organization’s MySQL operations requires in-depth knowledge of
potential issues plus diligent, dedicated practice. It is a vital competency if your business
depends on MySQL. Being aware of the issues above will help protect your
organization’s MySQL-based applications. It will also significantly enhance both
performance and scalability to deliver a better user experience.

Keeping abreast of best practices for preventing these dangers takes time and expertise.
If you are a busy MySQL DBA and you worry about the level of attention being given to
your databases, Percona can help.

Percona Support services are accessible 24x7 online or by phone to ensure your MySQL
is running optimally. We can also provide onsite or remote Percona Consulting for
current or planned projects, or in emergency situations. Every engagement is unique
and we will work with you to create the most effective solution for your business.

If you lack the resources to adequately manage your MySQL operations, you can
outsource full operations of your MySQL servers to Percona Managed Services. Percona
offers both outsourced database administration and backup and recovery services.

December 12, 2014 10

©Percona. All rights reserved.
 We can evaluate your current MySQL operations, your goals, and your resources to
collaboratively implement a solution that will:

 Prevent common operational problems

 Keep your MySQL performing optimally
 Lower your database operations and administration costs

Organizations that rely on Percona Managed Services benefit from our deep
operational expertise, 24x7x365 coverage from our worldwide team of experts, SLA
commitments, and unlimited incident response for a fixed monthly fee.

Percona Managed Services can support your existing database infrastructure. We

support all MySQL deployment options including hosted on premise, hosting at a
colocation facility, or purchased services from a cloud or database-as-a-service provider.
We also support all variants of MySQL.

The Percona Remote DBA Service provides full operational management of your MySQL
servers. We apply best practices to configure your servers, set up monitoring and alerts,
configure and manage your backups, respond to alerts 24x7x365, perform routine
operational tasks, and help plan for future growth. Our Remote DBA service allows you
to concentrate on enhancing your applications while we care for your databases.

The Percona Backup Service ensures that your backups are properly configured, run
successfully on schedule, and that any backup issues or production data recovery needs
are handled promptly by expert MySQL technicians. The system uses proprietary tools
and methods which reduce the cost and complexity of MySQL backup and recovery
through advanced data storage management. It also provides full visibility into the
status of all of your backups through a convenient online portal.

To learn about any of our services, please contact us at (208) 473-2904 or +44 208 133-
0309 in Europe or email us at sales@percona.com.

December 12, 2014 11

©Percona. All rights reserved.
 About Percona
Percona has made MySQL faster and more reliable for over
2,000 clients worldwide since 2006. Percona provides
enterprise grade MySQL Support, Consulting, and Managed
Services. Percona's founders authored the definitive book
High Performance MySQL from O'Reilly Press and the
widely read Percona MySQL Performance Blog. Percona
also develops software for MySQL users, including Percona
Server, Percona XtraBackup, Percona XtraDB Cluster, and
Percona Toolkit. The popular Percona Live conferences
draw attendees and acclaimed speakers from around the
world. For more information, visit www.percona.com.

December 12, 2014 12

©Percona. All rights reserved.