Professional Documents
Culture Documents
Wodp Jdlsjlas
Wodp Jdlsjlas
October 2013
Issue 19
by Alexander Rubin
Security holes? Who cares?
by Arne Blankerts
Percona Live London
Conference preview
Open Source Spotlight
Debugging tool Kint
In the key
of Symfony2
Music to my ears
Contents
This months top events
Open source spotlight: Kint
Interview with Rokas leinius
www.webandphp.com
3
4
Tutorial
by David Buchmann
Feature
Tutorial
14
Tutorial
18
Column
25
Ben Greenaway
Column
28
Column
Conference preview:
Percona Live London
30
Calendar
Community Corner
http://www.meetup.com/BellinghamDevelopers/events/133264492/
Following a bite to eat at Westside Pizza, AWS
evangelist Jeremy Lindblom will be giving a
talk on getting your app to run in the cloud.
http://webtuesday.ch/meetings/20131008/
Hear from Benjamin Regg, a developer for Swiss
newspaper Blick, on how he applies open source
tools to high-speed, large-scale environments. Talk
is in German.
http://www.meetup.com/PHPColMeetup/
events/129463532/
Learn about Facebooks custom replacement
to the Zend Engine, which can (apparently)
increase performance by 5x.
NoSQL Bakeoff
http://www.meetup.com/sawebdev/
events/135588782/
This months event is (at time of writing) still looking for volunteers to do a 1015 minute talk on
MongoDB, Neo4J, Redis or CouchDB.
www.webandphp.com
OSS Spotlight
Community Corner
www.webandphp.com
Homepage:
http://raveren.github.io/kint/
On GitHub:
https://github.com/raveren/kint/
by David Buchmann
www.webandphp.com
The CMF [2] aims to provide a way out of this dilemma. It consists of a couple of Symfony2 bundles
and PHP libraries that provide the means to do content
management. You can add them to a Symfony2 application to satisfy CMS needs. Or you can use them
as a base to build a completely custom CMS without
wasting time on the basics. Use the components that
make sense for the project, extend what you need and
ignore what you dont.
The CMF follows the idea of Decoupled Content
Management [3]. The Symfony2 bundles are about
functionality only. Persistence is cleanly separated,
and while a default handling for Doctrine PHPCRODM [4] is provided others can be used just as well.
The CmfCreateBundle enables frontend editing using
the general purpose create.js [5] editing framework.
The CMF is currently in release candidate phase,
with a stable 1.0 release expected at the time this
article is published. A very active community is maintaining and improving the various components. A couple of websites are already using the CMF or parts of
Listing 1
$ git clone git://github.com/symfony-cmf/cmf-sandbox.git
$ cd cmf-sandbox
$ cp app/config/parameters.yml.dist app/config/parameters.yml
$ cp app/config/phpcr_doctrine_dbal.yml.dist app/config/phpcr.yml
$ curl -s http://getcomposer.org/installer | php -$ ./composer.phar install
Listing 2
$ app/console doctrine:database:create
$ app/console doctrine:phpcr:init:dbal
$ app/console doctrine:phpcr:repository:init
$ app/console doctrine:phpcr:fixtures:load
Listing 3
<?php
namespace Sandbox\MainBundle\Document;
use Doctrine\ODM\PHPCR\Mapping\Annotations as PHPCRODM;
use Symfony\Cmf\Bundle\ContentBundle\Doctrine\Phpcr\StaticContent
class IntroContent extends StaticContent
{
/**
* @PHPCRODM\String(translated=true)
*/
private $intro;
// getter and setter for intro...
}
Figure 1: The CMF start page
Basic Usage
The best starting point to explore the CMF is the Sandbox [8]. You can click around in http://cmf.liip.ch , but
best is to install it locally to be able to change code.
You should now have a Symfony2 project with the
CMF set up. The storage engine we chose in Listing1
is Doctrine PHPCR-ODM with the SQLite database
driver. Next we need to initialize the database and the
repository. The last line of Listing 2 loads some fixtures so that we actually have something to look at.
Now you need to point a web server to the web/
directory of the project. When you point your browser
to http://cmf.lo/app_dev.php you will be greeted by the
default home page. Figure1 explains what CMF component handles which part of the page.
The CmfContentBundle handles simple static content. It provides a model class for the content and a
Symfony2 controller that uses either the specified or a
default twig template to render the content. The Cmf
MenuBundle extends the KnpMenuBundle, adding the
www.webandphp.com
Routing
When a request is received, the first thing Symfony2
has to do is to decide what code will handle this request. The main information used to make this decision is the requested URL, but other information such
as the host name used or accept headers can play a
role as well. This process is called Routing in Symfony2 and the result of the routing is information about
which Controller to call. In core Symfony2, routing is
configured using configuration files that define URL
patterns and corresponding controllers.
The CmfRoutingBundle extends Symfony2 core
routing to load routes from a database so that they can
be edited by the user. A CMF route can simply specify
a controller name. More interesting is the option to
reference a content object. An additional step called
Enhancer inside the routing processes the matched
route to infer additional information like the controller
and the template to use for the referenced content.
The aim of this design is to not duplicate configuration
Listing 4
# app/config/config.yml
cmf_routing:
dynamic:
# ...
templates_by_class:
Sandbox\Document\IntroContent: SandboxMainBundle:IntroContent:
index.html.twig
# ...
www.webandphp.com
the menu system, to get the URLs for menu items that
link to content.
To seamlessly integrate the CMF into standard
Symfony2 applications, the CmfRoutingBundle also
provides a ChainRouter which replaces the core Symfony2 router. The ChainRouter calls the core router,
the CMF router and any other routers that where registered to it, in a configurable order. That way, normal
Symfony2 routes are still possible and do not need to
be transferred into the database.
Editing
When the user is allowed to edit content, the CmfCre
ateBundle shows the Create.js toolbar at the top of the
page. By clicking on the Symfony2 logo, the editing bar
is minimized and expanded. Clicking the Edit field en
ables Frontend Editing. The content is edited right in the
location where it is shown. Create.js provides real WYSIWYG editing instead of changing the HTML structure by
adding an iframe around the editable content. That way,
the CSS does not need to be adapted and editing does
not diverge from display. Upon clicking Save, all edited content is stored over AJAX and the page is updated
without reloading. Figure3 shows the editing in action.
The CmfCreateBundle integrates create.js into Symfony2 with the help of the CreatePHP library [9]. Create.js uses RDFa attributes on content to know what
content is editable and how to communicate with
backend to save changes. CreatePHP implements the
backend handling for storing changes and also provides helpers so that template writers do not need to
manually handle the RDFa attributes. Sounds confusing? Lets add a new type of content to the sandbox to
see how this is all done.
Storage Layers
Though designed to be storage agnostic, there is a
preferred storage layer for the CMF: Doctrine PHPCRODM. This object-document mapper stores model
classes into the PHP content repository [10]. The reason for this is simply ease of use. PHPCR is tree-oriented, but inside the tree you can mix different kinds
of content. For a CMS, this means that content can
be structured according to its logic hierarchy rather
than by type. A CMS easily has a dozen or more types
of content that differ only on details. The other cru-
Listing 5
<!-- src/Sandbox/MainBundle/Resources/rdf-mappings/Sandbox.MainBundle.
Document.IntroContent.xml
xmlns:sioc="http://rdfs.org/sioc/ns#"
xmlns:dcterms="http://purl.org/dc/terms/"
xmlns:skos="http://www.w3.org/2004/02/skos/core#"
xmlns:sandbox="http://cmf.symfony.com"
typeof="sioc:Post"
>
<children>
<property property="dcterms:title" identifier="title" tag-name="h1"/>
<property property="sandbox:intro" identifier="intro" />
<property property="sioc:content" identifier="body" />
</children>
</type>
Listing 6
{% foreach pages as teaser %}
{% createphp teaser as rdf %}
<h1 class="my-title" {{ createphp_attributes( rdf.title ) }}>
{{ createphp_content( rdf.title ) }}
</h1>
<div class="intro" {{ createphp_attributes( rdf.intro ) }}>
{{ createphp_content( rdf.intro ) }}
</div>
{{ rdf.content|raw }}
{% endcreatephp %}
{% endfor %}
www.webandphp.com
David Buchmann is a core developer of the Symfony Content Management Framework and of the PHPCR and the Jackalope tool suite.
He works as software architect and PHP developer at the swiss
company Liip AG.
References
[1] www.symfony.com
[2] cmf.symfony.com
[3] http://decoupledcms.org/
[4] http://docs.doctrine-project.org/projects/doctrine-phpcr-odm/en/latest/
index.html
[5] http://createjs.org/
[6] http://cmf.symfony.com/references
[7] https://drupal.org/node/1874500
[8] https://github.com/symfony-cmf/cmf-sandbox
[9] http://createphp.org
[10] PHPCR was covered in a previous issue of Web & PHP: http://webandphp.
com/ThePHPContentRepository
Sylius
Feature
iStockphoto.com/cultury
www.webandphp.com
What is Sylius?
Sylius is a modern e-commerce platform based on
Symfony2 framework, and as an open source project,
announced under MIT license, it is therefore completely free for downloading and implementing any
changes. It is developed using BDD [4] and TDD methodology the only ecommerce solution implemented
using this methodology at the moment. It is characterized by a simple and clear architecture as well as
maximal flexibility. Even though it has not reached its
beta phase yet, developers are already using it for implementing complex and custom webshops.
Sylius is a new player among existing open source
ecommerce systems, such as Magento, osCommerce
and PrestaShop. Sylius is still not in beta release and
yet is already well-known within the Symfony2 community. Developers are using it for producing ecommerce solutions with high levels of customization.
What is very exciting is that you can participate in mak-
Sylius
Feature
Community
As mentioned earlier, Sylius is an open source project,
started by Pawe J drzejewski [5] in 2011. The idea of
commercializing the product existed at one point (KNP
Labs company), but Pawe gave up on the idea because
www.webandphp.com
Features:
yy Product Management: unlimited products, complex products with options and attributes, product
variations
yy Inventory tracking: real time tracking, products
available on demand, tracking of individual units
yy Complete taxation support: works with simple
and complex tax rules, rates for different tax categories and zones, tax included in price support,
customizable tax calculators
yy Product categories: tree structure, complex taxonomies, list your products by category, brand,
producer (and so on), create featured product
easily
yy Shipping engine: organize and track your shipments, unlimited number of shipping methods and
individual shipment units, track shipments, statuses, and individual shipment units, multiple flexible
shipping charge calculators
yy Easy to host: Syliys requirements are identical to
Symfony2 requirements, small setups run on cheap
hosting, easy to deploy with capifony (capifony
setup file comes with Sylius standard installation)
yy Free now and always: fully open source, business
friendly licence MIT; no paid edition with more
features
Architecture
Sylius was recently upgraded to the LTS Symfony version 2.3. (LTS = long term support for three
Sylius
years for any bugs and four years for any security issue)[7]. Sylius architecture is based on several decoupled components. At this moment, the standard
version is made of 19 different bundles [8]: AddressingBundle, CartBundle, CoreBundle, FlowBundle,
InstallerBundle, MoneyBundle, OmnipayBundle,
PaymentsBundle, ProductBundle, PromotionsBundle, ResourcesBundle, SalesBundle, ShippingBundle,
TaxationBundle, TaxonomiesBundle, VariableProductBundle, and WebBundle. Symfony also uses
some well known Symfony2 bundles as: FOSUserBundle, FOSRestBundle, KnpMenuBundle and
PagerFantaBundle.
The idea behind Sylius is to construct your own solution using the components, the same philosophy as
Symfony2. All features are also available as standalone
bundles.
Sylius is completely component-based, so shop
tasks such as payment, shipping, taxation, taxonomy
(and so on) are non-mandatory installs and not have to
necessarily have to come from Sylius itself. You can
write custom bundles, or you can use any other existing bundle. Thus, it introduces the idea of the Symfony2 framework early on, which has been integrated,
in this case, as a full stack.
After you have installed Sylius on your server, you
can do the complete customization through override
and customize your own interface. It is easy to customize core models and adapt the logic inside any service.
Also, it is possible to use any of the 1900 Symfony2
bundles in your Sylius project.
A part of the customization process is overriding the Twig templates in order to customize your
shop. Twig is a modern template engine for the
PHP programming language. It compiles templates
to optimized PHP code and the overhead compared
to regular PHP code was reduced to the minimum.
Also, Twig has a sandbox mode to evaluate untrust-
www.webandphp.com
Feature
Listing 1
curl -sS https://getcomposer.org/installer | php
composer create-project-s sylius dev / sylius
cd sylius
php app/console sylius:install
Sylius
Future plans
The major challenge is implementing/integrating a
payment system, since the main application does not
have any payment integration available out of the box.
The idea is to have a payment system as flexible as
other Sylius engines.
Pawe J drzejewski, who started Sylius, has started implementing a gateway-agnostic solution and
will share a POC soon. This should allow to integrate
any payment API library into Sylius and hopefully
there will be many integrations in the core. While
this feature is not implemented there is options for
using some of existing Symfony2 bundles like JMS
PaymentCoreBundle [13] (Apache 2.0 licensed, this
bundle is not maintained very actively, and it also requires using concrete data models but it is an an
interesting option anyway) or Omnipay [14]. An integration bundle for the Omnipay library has already
been created and it seems to work well with Sylius.
However, the library itself has some architectural issues and involves a lot of BC breaks. Therefore, one
must be careful when evaluating this option, but it
seems promising. Payum[15] at first glance looks like
the most complex option. It seems to be regularly
updated and the author has already approached the
community on GitHub. It certainly seems like a promising option.
In the latest major Sylius update, Cart and Order models have been merged. By doing this, the checkout-related logic has been dramatically simplified. It is important
to make sure that templates are displaying the data properly in the cart summary and the calculation logic and
events are already in place. If a user has selected the
shipping method, it should be displayed or it should even
be possible to allow it to be changed in the cart form.
Tax is recalculated on every cart change. However,
there is a case when the cart has changed and the user
has already picked the payment/shipping options, and
www.webandphp.com
Feature
Currently the FSCHateoasBundle [17] is used to handle the serialization of Pagerfanta (the paginator) instances. This bundle is more powerful and a more complete
integration and usage ought to be considered. Another
option is the BazingaHateoasBundle [18], a much younger solution which integrates a standalone library into
Symfony2. It seems to have all the features we need,
but both solutions should be considered in order to pick
the best one.
Sylius
www.webandphp.com
Feature
[3] http://symfony.com/blog/symfony2-meets-drupal-8
[4] http://en.wikipedia.org/wiki/Behavior-driven_development
[5] https://github.com/pjedrzejewski
[6] http://behat.org/
[7] http://sylius.org/blog/welcome-to-the-new-sylius-org
[8] http://twig.sensiolabs.org/
[9] http://sylius.org/blog/simpler-crud-for-symfony2
[10] https://github.com/Sylius/Sylius/tree/master/src/Sylius/Bundle
[11] https://demo.sylius.org
[12] http://www.twitter.com/Sylius
[13] https://github.com/schmittjoh/JMSPaymentCoreBundle
[14] https://github.com/adrianmacneil/omnipay
[15] https://github.com/payum
[16] https://github.com/FriendsOfSymfony/FOSRestBundle
[17] https://github.com/TheFootballSocialClub/FSCHateoasBundle
[18] https://github.com/willdurand/BazingaHateoasBundle
[19] https://github.com/Sylius/Sylius/issues/287
[20] http://getbootstrap.com/
[21] http://www.ymc.ch/en/sylius-a-great-piece-of-symfony2-for-ecommerce-but-is-it-ready-to-use
References
[1] www.symfony.com
[2] http://symfony.com/blog/symfony2-meets-ez-publish-5
Twig Tutorial
by Aurelio De Rosa
www.webandphp.com
Stand up if youve never used the amazing PHP construct include and/or require to assemble the layout
of a website ... I bet youre all still sat down. Me too.
Using PHP as a template engine is fine and works
well while youre building very small websites with
very few requirements. But what if you need more
flexibility? What about replacing parts of your layout
on demand? What if you want to dynamically include
other parts into the layout? Yes, all of this stuff is
possible with raw PHP too, but why reinvent the
wheel when you can rely on a fast, secure and flexible product?
I am, of course, referring to Twig. In this article Ill
show you what it is and how to use it to simplify your
work.
What is Twig?
Twig [1] is a template engine for PHP that has all the
major features you may need in your projects. It was
created by Fabien Potencier, the developer behind
Twig Tutorial
Installing Twig
One of the thing I appreciate is the wide range of possibilities to install Twig. The most common and recommended way is via Composer. The first step is to add it
to your projects dependencies as shown below:
...
require: {
...
twig/twig: 1.*
}
...
or run this:
php composer.phar install
An alternative method to obtain Twig is via PEAR, typing the following commands:
pear channel-discover pear.twig-project.org
pear install twig/Twig
You can obtain Twig in many other ways but Ill present just another one, the simplest one: downloading
the files from the download page of the project on
GitHub [3]. Grab the last release (1.13.2 at the time
of writing), extract the files from the archive and put
them into your projects folder.
www.webandphp.com
Listing 1
{# Im a comment and these are some examples #}
{# Evaluates the variable bookTitle and print the result inside an h1 #}
<h1>{{ bookTitle }}</h1>
{# Evaluates the expression composed of the authorName variable and the string
"De Rosa", concatenated by the "~" character #}
<h2>{{ authorName ~ "De Rosa" }}</h2>
{# Evaluates the variable count and change the resulting template accordingly #}
{% if count < 1000 %}
<p>This is a good book</p>
{% else %}
<p>This is a best seller</p>
{% endif %}
In the code above I showed the use of simple variables which can store numbers, strings and similar. In
Twig, as you might expect, you can also deal with array
and objects. To use their values, youve to use a Java
Script-like syntax, therefore you can access a property
using the dot operator (for example obj.prop) or using
the square brackets (for example obj['prop']). Keep in
mind that the previous methods can be used when an
attribute contains a Twigs special characters like the
hyphen, that is interpreted as the minus operator. In
these cases, you have to use the attribute() function
like this: attribute(obj, 'prop')
Tags
So far, youve seen how to use variables and expressions, so its time to delve into the second concept of
Twig templates: tags. As said, they give you control
over the template and how it acts. In the previous examples youve already seen a couple of tags, that is
Twig Tutorial
set and if. Now, what if you want to print a list of items,
for example a list of ingredients of a recipe? This is
exactly where the for tag comes into play. To perform
this task, you can do something like:
<h1>Ingredients</h1>
<ul>
{% for ingredient in ingredients %}
<li>{{ ingredient }}</li>
{% endfor %}
</ul>
www.webandphp.com
Filters
You probably find yourself often modifying variables
in a PHP page using functions for example, escaping HTML characters or converting a sentence to title case. Doing it in the controllers (or in the page
if you arent adopting an MVC pattern) isnt a really
good practice, because if the same controller is used
for other purposes you may need not to apply those
functions. The best place to apply those filters is
in the view and luckily Twig knows this, providing you with a bunch of them. Filters are separated
from the variable by a pipe symbol (|) and may have
optional arguments closed between two brackets.
In addition, you can apply multiple filters by chaining
them. Recalling the example I outlined, this is a snippet that takes a variable called bookTitle, escapes
HTML characters and capitalizes the first letter of
each word:
Listing 2
<!DOCTYPE html>
<html>
<head>
{% include "meta.twig" %}
<!-- asserts here... -->
</head>
<body>
<!-- layout here... -->
</body>
</html>
Listing 3
<!DOCTYPE html>
<html>
<head>
<title>{% block pageTitle %}{% endblock %}</title>
<!-- asserts here... -->
</head>
<body>
<head>
<img src="site-logo.jpg" alt="" />
<h1>Aurelio De Rosas website</h1>
</head>
<main>
{% block content %}{% endblock %}
</main>
</body>
</html>
{{ bookTitle|escape|title }}
Twig Tutorial
www.webandphp.com
Conclusions
In this article we discovered what Twig is and its basic components. Throughout it, we uncovered the key
features of the framework, like variables, tags, and
filters, and also studied several examples that should
help using it easily in your next projects. Keep in mind
that the library has also some important extensions not
described here, like the I18N and Intl, which are very
useful if youre developing a multilingual project. With
this last hint, the article is over. I hope you enjoyed it
and that thanks to it youll consider using Twig or another of the available template frameworks.
Aurelio De Rosa is an Italian web and app developer with more than
5 years experience programming for the web using HTML5, CSS3,
JavaScript and PHP. He mainly uses the LAMP stack and frameworks
like jQuery, jQuery Mobile, and Cordova (PhoneGap) but his interests
also include web security, web accessibility, SEO, WordPress and
everything he can put his hands on. In the little spare time left, he contributes
to the open source community writing libraries for PHP as well as jQuery plug
ins. Currently hes self-employed working with the cited technologies. Hes also
a regular blogger for several networks (SitePoint, Tuts+, FlippinAwesome) where
he writes articles about the topics he usually works with and more.
References
[1] http://twig.sensiolabs.org/
[2] http://www.smarty.net/
[3] https://github.com/fabpot/Twig/tags
[4] http://twig.sensiolabs.org/doc/tags/index.html
[5] http://twig.sensiolabs.org/doc/filters/index.html
[6] http://twig.sensiolabs.org/documentation
Indexes
In this article we will focus on the b-tree indexes only
(InnoDB and MyISAM only support B-tree indexes). Figure1 illustrates a basic b-tree index [1] implementation.
by Alexander Rubin
www.webandphp.com
Explain plan
The main way to profile a query with MySQL is to
use explain [3]. Listing2 shows an example. As we
can see from the explain, MySQL does not use any
indexes (key: NULL) and will have to perform a full
table scan.
In this case we can add an index to restrict the number of rows:
Index usages
MySQL will choose only 1 index per query (and per
table if the query joins multiple tables). In some cases
Listing 1
CREATE TABLE City (
ID int(11) NOT NULL AUTO_INCREMENT,
Name char(35) NOT NULL DEFAULT '',
CountryCode char(3) NOT NULL DEFAULT '',
District char(20) NOT NULL DEFAULT'',
Population int(11) NOT NULL DEFAULT '0',
PRIMARY KEY (ID),
KEY CountryCode (CountryCode)
) Engine=InnoDB;
Listing 2
mysql> EXPLAIN select * from City where Name = 'London'\G
*************************** 1. row
id: 1
select_type: SIMPLE
table: City
type: ALL
possible_keys: NULL
key: NULL
key_len: NULL
ref: NULL
rows: 4079
Extra: Using where
www.webandphp.com
Combined Indexes
Combined indexes are very important for MySQL query
optimizations. MySQL can use leftmost part of any index.
For example, if we have this index: Comb(CountryCode,
District, Population). Then MySQL can use:
CountryCode only part
CountryCode + District
CountryCode + District + Population
From the explain plan in Listing 4 we can understand
which part(s) will be used. Note the key_len part
it shows: 3. This is the number of bytes used from
our index. As the CountryCode field is declared as
char(3), that means that MySQL will use the first
field from our combined index.
Similarly, MySQL can use 2 leftmost fields or all 3
fields from the index. In Listing5, the 2 leftmost fields
Listing 3
mysql> explain select * from City where Name = 'London'\G
*********************** 1. row ***************************
id: 1
select_type: SIMPLE
table: City
type: ref
possible_keys: Name
key: Name
key_len: 35
ref: const
rows: 1
Extra: Using where
Covered index
The covered index is an index that covers all fields in
the query. For example, for this query:
Listing 4
mysql> explain select * from City
where CountryCode = 'USA'\G
********************** 1. row ******************
table: City
type: ref
possible_keys: comb
key: comb
key_len: 3
ref: const
rows: 273
select name from City where CountryCode = 'USA' and District = 'Alaska' and
population > 10000
Listing 5
mysql> explain select * from City
where CountryCode = 'USA' and District = 'California'\G
********************** 1. row ******************
table: City
type: ref
possible_keys: comb
key: comb
key_len: 23
ref: const,const
rows: 68
Listing 6
mysql> explain select * from City
where CountryCode = 'USA' and District = 'California'
and population > 10000\G
********************** 1. row ******************
table: City
type: range
possible_keys: comb
key: comb
key_len: 27
ref: NULL
rows: 68
www.webandphp.com
Listing 7
mysql> explain select * from City where
District = 'California' and population > 10000\G
********************** 1. row ******************
table: City
type: ALL
possible_keys: NULL
key: NULL
key_len: NULL
ref: NULL
rows: 3868
Listing 8
mysql> explain select name from City where CountryCode = 'USA' and District =
'Alaska' and population > 10000\G
*************************** 1. row ***********
table: City
type: range
possible_keys: cov1
key: cov1
key_len: 27
ref: NULL
rows: 1
Extra: Using where; Using index
1. Where part
2. Group By/Order (not used now)
3. Select part (here: name)
You can see an example of this in Listing8. Using index in the extra field of the explain output means that
MySQL will use our covered index. That also means
that MySQL will use an index only to satisfy the query:
all the information that MySQL needs is in the index.
That is usually much faster, especially if we have a
large text or blob fields in the table.
Listing 9
mysql> explain select name from City where CountryCode = 'USA' and District =
'Alaska' and population > 10000\G
*************************** 1. row ***********
table: City
type: range
possible_keys: cov1
key: cov1
key_len: 27
ref: NULL
rows: 1
Extra: Using where; Using index
Listing 10
...
type: ALL
possible_keys: NULL
key: NULL
key_len: NULL
ref: NULL
rows: 4833086
Extra: Using where; Using temporary; Using filesort
Listing 11
mysql> explain select name from City where CountryCode = 'USA' and District =
'Alaska' and population > 10000\G
*************************** 1. row ***********
table: City
type: range
possible_keys: cov1
key: cov1
key_len: 27
ref: NULL
rows: 1
Extra: Using where; Using index
www.webandphp.com
Group by example
Lets look at this simple example, a query of how
many cities there are in each country. As we can see
from Listing9, MySQL does not use any indexes (no
proper indexes are available), but it also shows Using temporary; Using filesort. MySQL will need to
create a temporary table to satisfy the Group by
clause if there is no appropriate index (You can find
more information about the temporary tables at [4]).
However, MySQL can use a combined index to satisfy the group by clause and avoid creating temporary table.
Group by and covered index
To illustrate the group by queries I will use the following table:
Listing 12
mysql> alter table ontime_2012
add key covered(dayofweek, Carrier, DepDelayMinutes);
explain select max(DepDelayMinutes), Carrier, dayofweek from ontime_2012 where
dayofweek =7 group by Carrier\G
...
possible_keys: DayOfWeek,covered
key: covered
key_len: 2
ref: const
rows: 905138
Extra: Using where; Using index
Listing 13
mysql> explain select max(DepDelayMinutes), Carrier, dayofweek from ontime_2012
where dayofweek > 5 group by Carrier, dayofweek\G
...
type: range
possible_keys: covered
key: covered
key_len: 2
ref: NULL
rows: 2441781
Extra: Using where; Using index; Using temporary; Using filesort
Listing 14
(select max(DepDelayMinutes), Carrier, dayofweek
from ontime_2012
where dayofweek = 6
group by Carrier, dayofweek)
union
(select max(DepDelayMinutes), Carrier, dayofweek
from ontime_2012
where dayofweek = 7
group by Carrier, dayofweek)
from ontime_2012
where dayofweek = 7
group by Carrier
Listing 15
*************************** 1. row ***************************
table: ontime_2012
key: covered
...
Extra: Using where; Using index
*************************** 2. row ***************************
table: ontime_2012
key: covered
...
Extra: Using where; Using index
*************************** 3. row ***************************
www.webandphp.com
Listing 16
id: NULL
select_type: UNION RESULT
table: <union1,2>
type: ALL
possible_keys: NULL
key: NULL
key_len: NULL
ref: NULL
rows: NULL
Extra: Using temporary
Figure 6: Speed of queries with temporary table, tight index scan (covered index)
and loose index scan
www.webandphp.com
Benchmark
We have performed a benchmark to compare query
speed with a temporary table, a tight index scan (covered index) and a loose index scan. The table is 6 million rows and approximately 2GB in size.
The results are shown in Figure6. As we can see,
loose index scan index shows the best performance.
Unfortunately, loose index scan only works with 2 aggregate functions, min and max for the group by. AVG
+ group by does not work with loose index scan. As we
can see in Listing16, MySQL will use a covered index
(not loose_index_scan index) and, because we have a
range in the where clause (dayofweek > 5), it will have
to create a temporary table.
ORDER BY and filesort
MySQL may have to perform a filesort operation when
a query uses the order by clause. You can find more
information about filesort at [7].
The filesort operation in Listing19 is usually pretty
slow operation (even if it does not involve creating a
file on disk), especially if MySQL will have to sort a lots
of rows.
To optimize this query we can use a combined index
(Listing 20). MySQL was able to use our combined
index to avoid sorting: as the index is sorted, MySQL
was able to read the index leafs in the correct order.
Sorting and limit
If we have a LIMIT clause in our query (and the
limit is relatively small (i. e. LIMIT 10 or LIMIT 100)
Listing 17
mysql> explain select max(DepDelayMinutes) as ddm, Carrier, dayofweek from
ontime_2012
where dayofweek > 5 group by Carrier, dayofweek;
table: ontime_2012
type: range
possible_keys: covered
key: loose_index_scan
key_len: 5
ref: NULL
rows: 213
Extra: Using where; Using index for group-by;
Listing 18
mysql> explain select avg(DepDelayMinutes) as ddm, Carrier, dayofweek from
ontime_2012 where dayofweek >5 group by Carrier, dayofweek \G
table: ontime_2012
type: range
key: covered
key_len: 2
ref: NULL
rows: 2961617
Extra: Using where; Using index; Using temporary; Using filesort
Listing 19
table: City
type: ALL
possible_keys: NULL
key: NULL
key_len: NULL
ref: NULL
rows: 4079
Extra: Using where; Using filesort
Listing 20
mysql> alter table City
add key my_sort2 (CountryCode, population);
mysql> explain select district, name, population from City where CountryCode =
USA order by population desc limit 10\G
table: City
type: ref
key: my_sort2
key_len: 3
ref: const
rows: 207
Extra: Using where
Conclusion
We have discussed different indexing strategies to optimize your slow queries:
Listing 21
mysql> explain select * from ontime_2012
where dayofweek in (6,7) order by DepDelayMinutes desc
limit 10\G
type: index
possible_keys: DayOfWeek,covered
key: DepDelayMinutes
key_len: 5
ref: NULL
rows: 24
Extra: Using where
References
[1] http://en.wikipedia.org/wiki/B-tree
[2] http://dev.mysql.com/doc/index-other.html
[3] http://dev.mysql.com/doc/refman/5.5/en/explain.html
[4] http://dev.mysql.com/doc/refman/5.5/en/internal-temporary-tables.html
[5] http://www.transtats.bts.gov/DL_SelectFields.asp?Table_ID=236&DB_
Short_Name=On-Time
[6] http://dev.mysql.com/doc/refman/5.5/en/loose-index-scan.html
[7] http://dev.mysql.com/doc/refman/5.5/en/order-by-optimization.html
www.webandphp.com
OSS Column
Creating business
tools in cyberspace?
Think Different!
by Ben Greenaway
www.webandphp.com
It may not be obvious why, but I consider WordPress as belonging to these Software as a Service
offerings. Im sure you are familiar with it as a free
and open source blogging tool, and to many small
business owners it is also an essential piece of
their communications and relationship management
strategy. Modern business requires good customer
service, sales require brand awareness and marketing campaigns need mass distribution all of which
have significant social networking components
these days. Its hard to look like a modern business
without one.
Bio
Ben Greenaway has been a software developer
for 18 years. After pioneering internet narrowcasting in the 90s in collaboration with Virtual
Futures he has led development on A/V installation projects for Pixar and Sony Entertainment
and web applications for SMEs in Southern
California before moving to the San Francisco
Bay in 2010 and developing performance solutions for Dyn DNS and e-commerce APIs for
Tzolkin TZO. He now lives in South London
where he balances time writing and developing
with the demands of an ever growing collection
of retro computers.
OSS Column
Available
now for
$ 3.99
Buy now at:
www.developerpress.com
Also available to buy on:
www.webandphp.com
OSS Column
www.webandphp.com
Security Column
Security holes?
Who cares?
by Arne Blankerts, thePHP.cc, Germany
Bio
Every now and then, things go wrong. This is how
life is and there is probably nothing we can really do
about it. On the other hand, we automate more and
more things, have computers and machines take
over to reduce the potential of humans making mistakes. Happily ignoring the fact all those machines
are built and programmed by humans, we blindly
trust the device at hand. And if things go wrong anyway, its not a human mistake but a computer glitch.
www.webandphp.com
Arne Blankerts consults for thePHP.cc, solving IT problems long before many companies
realize that they even exist. IT security is his
passion, which he pursues with almost magical intuition, creating solutions that always
bear his hallmark. Companies around the
world rely on his concepts and Linux-based
system architectures.
Security Column
www.webandphp.com
References
[1] http://venturebeat.com/2013/08/18/facebook-zuckerberg-wallvulnerability/
When comparing the various versions and iterations of said lists, one thing stands out: Even if the
order of the items changes, there seems to be hardly
any real change in terms of content. The standard
mistakes keep being ranked high in all lists. And
[2] http://www.vulnerability-lab.com/dev/index.php/2013/04/29/5000-paypalbug-bounty-auth-bypass-vulnerability/
[3] http://www.telegraph.co.uk/finance/newsbysector/industry/10211760/
Volkswagen-sues-UK-university-after-it-hacked-sports-cars.html
[4] http://www.sans.org/top25-software-errors/
[5] https://www.owasp.org/index.php/Top_10_2013-Top_10
by
Conference preview:
Percona Live London
Book your ticket to hear from the likes of MySQL and MariaDB creator Monty
Widenius and VP MySQL Engineering Tomas Ulin on November 1112, 2013
www.webandphp.com
f ticketsof!
o
r
i
a
p
a
air
Win
ill win a p
w
n
y reader
t. Send a
One luck
er's even
b
m
e
v
m
o
o
N
andphp.c
tickets to
tb@web
io
na Live
ll
o
e
rc
e
to
P
"
il
ema
) titled
5
1
r
e
b
to
, Oc
e to win.
(by 11pm
ur chanc
o
y
r
fo
"
tion
Competi
www.webandphp.com
size. About a third of the attendees came from companies with fewer than 50 employees, and about a
fifth from companies with more than 501 employees.
The vertical focus of the represented companies also
covered a wide range, including software, telecommunications, e-commerce, media, retail, entertainment,
mobile technology, education, hospitality, social networking, biotechnology, and healthcare.
While the Percona Live MySQL Conference in London is a chance to play an active role in a growing community of professionals, it also offers specific benefits
for three groups of attendees.
For MySQL professionals in Europe, the conference
is the best opportunity to connect with customers,
partners, and other stakeholders in the MySQL ecosystem. Because most of our day-to-day contact may
be online, the conference can be a great opportunity to
meet face-to-face, nurture relationships, and explore
the possibility of future collaboration.
www.webandphp.com
References
[1] http://lists.mysql.com/mysql/228795
[2] https://www.percona.com/live/london-2013/
[3] https://www.percona.com/live/london-2013/conference-committee
[4] https://www.percona.com/live/london-2013/
About
Publisher
Software & Support Media Ltd
Editorial Office Address
86 Great Suffolk St
London SE1 0BE
www.sandsmedia.com
Editor: Elliot Bentley (elliotb@sandsmedia.com)
Authors: Arne Blankerts, David Buchmann, Aurelio De Rosa,
Terry Erisman, Ben Greenaway, Antonio Peri -Maar,
Alexander Rubin, Rokas leinius,
Creative Director: Jens Mainz
Layout: Tobias Dorn, Flora Feher, Petra Rth
Sales:
Ellen May
+44 207 199 6234
ellenm@sandsmedia.com
Contents copyright 2013 Software & Support Media Ltd.
All rights reserved. No part of this publication may be reproduced, redistributed, posted online or reused by any means in
any form, including print, electronic, photocopy, internal network,
Web or any other method, without prior written permission of
Software & Support Media Ltd.
The views expressed are solely those of the authors and do
not reflect the views or position of their firm, any of their
clients, or Publisher. Regarding the information, Publisher disclaims all warranties as to the accuracy, completeness, or adequacy of any information, and is not responsible for any errors,
omissions, inadequacies, misuse, or the consequences of using any information provided by Publisher. Rights of disposal of
rewarded articles belong to Publisher. All mentioned trademarks
and service marks are copyrighted by their respective owners.