Magento Performance Testing For High Performance

MAGENTO HOSTING
Best Practices for Optimum Performance v1.6

Copyright 2000 2013 Nexcess.net LLC. All Rights Reserved.
This work is licensed under aCreative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.
April 2013
Brad Boegler magento-whitepaper@nexcess.net
Magento Hosting Best Practices for Optimum Performance
2
Table Of Contents
Introduction 3
Testing Criteria 4
Terms and Defnitions 4
Benchmarking Tools 5
Software Tools & Plug-ins 5
Benchmarking Parameters 6
Benchmarking Hardware Architecture 7
Software and Benchmarking Confgurations 9
Software Versions 9
Anatomy of a Siege Test Raw Stress Testing 9
Anatomy of a Gatling Test User Modeling Stress Testing 9
The Test Cases 11
Test 1: File Synchronization NFS vs. Local File System 11
Test 1: Results 11
Test 2: Apache vs. Nginx 12
Test 2: Results 13
Test 3: MySQL versus Percona 15
Test 3: Results 15
Optimized Magento Site versus Optimized Magento Site with Varnish & Turpentine 17
Test 4: Best confguration versus Varnish & Turpentine 17
Result Observations and Optimal Confgurations 20
The Importance of the Magento Enterprise Full Page Cache 20
The Magento DEFAULT_LIFETIME 21
Response Times 23
Understanding Magento Caching 25
Magento Backend Cache 25
Additional improvements 26
Magento Full Page Cache 26
Magento Session Cache 26
PHP opcode cache 26
Conclusion and Additional Performance Recommendations 27
Appendix A: System Confgurations 28
Appendix B: Benchmarking Confgurations 28
Gatling Tool 28
Siege 28
3
Introduction
Magento is a very popular and powerful eCommerce platform with very demanding resource requirements.
While it will run out of the box using default confguration settings, extensive post-installation confguration is
recommended to achieve optimal site performance (especially for high volume sites).
This white paper is meant as a guide for you to ensure youre getting the most out of your Magento installation
by using best practices developed through real-world experience at Nexcess, where weve been working with
the platform since 2007. We outline the diferences between default and/or popular available confgurations,
listing the benefts and drawbacks of these changes, and what weve found to be optimal for scaling Magento.
These diferent confgurations are extensively benchmarked showing the benefts between confguration op-
tions and the maximum performance available from each chosen setting.
Using the confguration information contained within this white paper, a load-balanced web server cluster run-
ning Magento Enterprise is able to process over 550 transactions per second (equating to more than 47 million
transactions per day). Further performance is obtained by utilizing the Nexcess Turpentine plug-in (with the
Varnish reverse proxy), which can provide up to 6,940 transactions per second (over 600 million transactions per
day).
All system-level confguration fles are provided at the end of this document and show the exact system settings
and application confguration parameters used. All benchmark application test confgurations are also provided
to show exactly how each test was run and how, if necessary, each can be reproduced.
We welcome your input and feedback. Any questions or comments may be directed to magento-whitepaper@
nexcess.net.
4
Testing Criteria
Terms and Defnitions
The following terminology can at times be confusing in relation to the actual process occurring. The terms used
throughout this white paper are defned as follows:
Buyer
A client who completes an order.
Client
A user and (more specifcally) the browser and/or browser mechanism used by the user. It does not, for the pur-
poses of this paper, always denote the customer of an eCommerce transaction.
Order
The successful completion of the checkout process within Magento.
Orders per second (also denoted as O/s or Orders/sec)
The number of orders processed during one second.
Request
Any HTTP communication sent by a client (browser, etc.) to a Magento server. This includes requests sent over
HTTPS and is synonymous with an HTTP request.
Requests per second (also denoted as R/s or Requests/sec)
The amount of requests being sent to a Magento server during one second. This metric does not take into ac-
count how many requests result in a successful response from the server (that is, a complete transaction).
Response
Any HTTP communication sent by a Magento server to a client. This includes those requests sent over HTTPS and
is synonymous with an HTTP response. Bear in mind, responses are not synonymous with a web page. A single
web page may have hundreds of individual responses associated with it, including images, HTML pages, style
sheets, JavaScript and other static or dynamic content.
Transaction
The combination of a successful request and response.
Transactions per second (also denoted as T/s or Transactions/sec)
The number of transactions completed during one second.
User
Synonymous with a client (for the purposes of this paper).
Visitor
A client who simply browses a website and does not complete an order.
5
Benchmarking Tools
Two benchmarking tools, Siege and the Gatling Stress Tool, were used extensively in testing.
Siege is an open-source, HTTP-based benchmarking utility which excels in obtaining the raw load capabilities
of a website. It can stress a web address (or a list of web addresses) with various concurrency levels. It is an ideal
load tester for determining the peak transactions per second for an individual client and the specifc response
times experienced.
The Gatling Stress Tool is also an open-source tool and is used to benchmark interactive features of a site. It can
be confgured to perform complex multi-step actions such as adding items to a cart and performing a checkout
in the same manner a human client would. Because of this, Gatling is excellent for testing the number of orders
per second a site is capable of processing under various load situations.
Software Tools & Plug-ins
The following software and other tools are mentioned throughout this white paper and are important enough
to the performance of Magento to merit a brief explanation:
Alternative PHP Cache (APC)
An opcode cache for PHP. When PHP processes a script it has to digest the PHP code into an intermediate format
that can be run by the PHP interpreter. Much of the work done on this digestion can be re-used in future exe-
cution of the script by saving some metadata about the script in the form of opcodes. This can speed up each
PHP scripts processing time by saving some time associated with compilation. APC can also be used as a mem-
ory-based caching system to cache key/value pairs in memory much like memcached (see description below),
though it sufers from the same defciencies.
memcached
A memory-based caching system. By using a simple key/value system to store data in a table in memory, access
to the data in memcached is very fast. However, a disadvantage to its implementation is no targeting is used
when fushing the cache. A cache fush is an all-or-nothing operation which can be non-optimal if the bulk of the
cache is not dirty. memcached uses a threaded model, allowing it to process requests in parallel efciently.
PHP FastCGI Process Manager (PHP-FPM)
This allows PHP processes to be pooled and reserved on a per-user basis. Traditional execution of the PHP inter-
preter involved starting up an entirely new process to handle a given request via a mechanism like CGI. PHP-FPM
allows for having many PHP processes running, which greatly enhances performance. Beyond this, PHP-FPM
has the added security beneft of allowing the PHP processes themselves to be assigned to a given user (which
allows user segmentation not available when using traditional Apache+mod_php).
6
Redis
A memory-based caching system. It uses a simple key/value system to store data in a table in memory (much like
memcached). Redis adds the advantage of tags to the storage model, which allows an application using Redis
to target any dirty sections of the cache during fush operations. Its current implementation is single-threaded,
meaning it can only process requests serially (that is, one at a time) for each instance of Redis.
Varnish
A web application accelerator (also known as a caching HTTP reverse proxy). Varnish can be used to drastically
speed up web applications with minimal performance degradation when confgured correctly. Varnish stores
the assembled web pages (or parts of web pages) in memory and relieves the application itself from having to
perform any work when a cache hit is realized. The Nexcess Turpentine Magento extension makes use of Varnish
to greatly enhance the performance of Magento Enterprise. By default, Varnish does not work well with Magen-
to because it doesnt cache requests with cookies. Magento sends a frontend cookie with every request caus-
ing a (near) zero hit-rate for Varnishs cache. Turpentine confgures Varnish to work with Magento and modifes
Magentos behavior to signifcantly improve the cache hit rate.
Benchmarking Parameters
Since every Magento store is diferent and the amount of data housed within one can make a big diference
when it comes to performance, weve standardized on the following Magento confguration for this white paper:

Magento version: Enterprise 1.12.0.2
Stores: 1
SKU count: 100,000 (thanks to our friends at Magento for providing this pre-built data set)
Categories: 10
Subcategories: 40
Some standard confgurations are set throughout the testing as they are best practices for Magento. These con-
fgurations are:
Two level Magento backend cache is enabled.
First level is set for memcached.
Second level is set to Redis with the CM_Cache plug-in.
The two-level backend cache patch is applied.
Sessions are stored in a dedicated memcached instance.
The Magento compiler is enabled.
The Magento full page cache is enabled (with the storage set to a second Redis instance).
The APC PHP opcode cache is used.
7
Benchmarking Hardware Architecture
A Nexcess MCE-SIP-200 server cluster was used for benchmarking in all of our test runs. This cluster consists of
a frewall, two web application servers, a single database server, a dedicated load balancer, and a dedicated fle
server (which also shares a role as a Magento application caching server).
Web Application
Server 1
Web Application
Server 2
Master Database
Server
Dedicated Cluster
File Server and
Caching Server
Dedicated Load
Balancer
Dedicated Managed
Firewall
Incoming requests
Outgoing Trafc
Internal File and Cache Trafc
Internal Database Trafc
Internet
Nexcess Redundant
Internal 10GbE
Network
Diagram 1: Nexcess MCE-SIP-200 Cluster Architecture
The Nexcess MCE-SIP-200 cluster used for testing consisted of the following hardware:
Web Application Servers:
Dell R420
2x Intel Xeon 8 Core CPUs E5-2450 @ 2.10GHz
24GB RAM
2x 147GB 15K SAS RAID-1 (H710)
Database Server:
Dell R420
24GB Ram
4x 147GB 15K SAS RAID-10 (H710)
8
File Server / Caching Server:
Dell R420
24GB Ram
4x 147GB 15K SAS RAID-10 (H710)
Load Balancer:
Dell R420
16GB Ram
Benchmarking Server (from which all tests were run):
Dell R420
24GB Ram
The testing server was placed across a gigabit network which was connected from within the Nexcess data cen-
ter in Dearborn, MI to the public interfaces of the testing cluster.
9
Software and Benchmarking Confgurations
Software Versions
The following software versions were installed on all servers in the MCE-SIP-200 cluster used for testing:
CentOS 6.3 (x86_64) Percona 5.5.28
PHP 5.3.17 MySQL 5.5.29-1
APC 3.1.9-5 memcached 3.0.6-3
Redis 2.4.10-1 Apache 2.2.23 (pre-fork MPM)
Nginx 1.0.15-3 PHP53-fpm-5.3.17-2
Varnish 3.0.2 Nexcess Magento-Turpentine 0.3.1
Furthermore, the benchmarking server ran the following software versions:
Siege 2.72 Gatling Tool 1.4.1
Anatomy of a Siege Test Raw Stress Testing
A Siege test was run on each specifc confguration for 60 seconds and then repeated once per minute over 100
minutes (for a total of 100 tests). Data was collected after each 60 second run. In addition to the 100 minute tests,
additional tests were run for a duration of 240 minutes each to observe any long term behavior of the Magento
confgured environment.
To ensure a broad spectrum of targets, we created a static list of pages on the test site using a method popular-
ized by Ashley Schroder on his site, magespeedtest.com. Utilizing the sitemap.xml from the data sets provided
by Magento on the active test site, we were provided with over 100,000 pages (including every product and
category page) to test. Details of sitemap-based testing can be found at http://www.aschroder.com/2010/12/lay-
ing-siege-to-magento-performance-testing-your-magento-webstores/.
Each test was run fve times against a particular confguration. The test results were then averaged. Before each
complete run, all relevant Magento caches were fushed and then pre-warmed by having Siege hit every URL in
our test fle exactly once. The test fle consisted of a complete Magento sitemap.
Anatomy of a Gatling Test User Modeling Stress Testing
We used Gatling to simulate two basic scenarios:
1. A buyer in the following use case (in sequence):
a. Navigate to the home page.
b. Navigate to a category page.
c. Navigate to a product page.
d. Add product to the cart.
e. View the cart.
f. Proceed through checkout process fully using the check/money-order payment method.
10
2. A visitor in anonymous catalog navigation (in sequence):
a. Navigate to the home page.
b. Navigate to a random category page.
c. Navigate to a random product page.
The stress tests were confgured to simulate both visitors and buyers with a 100:1 ratio. Thus, given 2,000 visitors,
there would be roughly 20 buyers. The intention behind the Gatling test was to place real world stress on a full
Magento environment with actual orders being placed. Simulating buyers engages parts of the system, such as
database bound requests, that are much harder to simulate with a simple Siege test. This allowed us to fnd the
optimal performance parameters for more realistic trafc profles.
Each test was run fve times against a particular confguration. The test results were then averaged. Before each
complete run, all relative Magento caches were fushed so each new confguration to be tested would begin
with an unpopulated and empty, or cold, fast backend, slow backend, session, and full page cache.
11
The Test Cases
Test 1: File Synchronization NFS vs. Local File System
File synchronization is a necessity when running multiple web application servers with Magento. Both the code
and metadata within Magento should be synchronized across all web application servers in order for it to func-
tion properly.
In this test we compared a pure NFS solution against a hybrid NFS/local store solution. NFS (or any shared stor-
age engine) plays a key role in ensuring certain metadata is synchronized in real time. Thus, for the sake of this
test, the media directory is always shared across all web application servers using NFS.
We tested two separate options for fle synchronization between the web application servers. The frst used NFS
for all synchronization and the second used the local fle systems on each web application server to serve con-
tent. When using a local store method, we provided fle synchronization via rsync.
There are both advantages and disadvantages to each confguration option. While NFS ofers greater simplicity
of site management (any code changes are immediately propagated to the web application servers) there is
some performance overhead required. This overhead shows up in the benchmarking results. The local fle sys-
tem performs better as long as the local disk I/O subsystem on each server is properly designed using fast SAS or
SSD drives and a good caching controller.
There is one additional comment regarding the shared media directory over NFS: When used with a CDN for
distributing static content with images, most images will come from the CDN and not the NFS-mounted fle sys-
tems. This prevents further performance degradation from using NFS while maintaining Magento administrative
functionality.
Test 1: Results
This test was implemented utilizing Siege with 250 concurrent users.
12
0
100
200
300
400
500
600
T=1 T=10 T=20 T=30 T=40 T=50 T=60 T=70 T=80 T=90 T=100
Local File System vs. NFS Transactions Per Second
NFS File System Local File System
Time in Minutes
T
r
a
n
s
a
c
t
o
n
s

p
e
r

S
e
c
o
n
d
Diagram 2: Local File System vs. NFS Transaction Per Second
Running Magento on a local fle system resulted in an average of 554 transactions per second while an
NFS-mounted fle system resulted in an average of 500 transactions per second. This accounts for a roughly 11%
performance increase and is consistent regardless of the web server software used.
The choice of whether to run Magento from either an NFS fleserver or locally on each web node comes down to
convenience and ease of usability of the site confguration as opposed to gains to be made in performance. NFS
ofers the beneft of your site fles existing in only one location. When making changes to the site, or moving a
site confguration from development or staging to production, only one location must be modifed. Using NFS,
these changes will be instantly visible on each web application server. On the other hand, using local fle systems
requires each web node to contain an entire copy of the Magento site and confguration. The copy then must
be kept in synchronization to the fleserver via rsync or other means. In this confguration, the Magento media
directory will still be shared across NFS. The Magento admin functionality would otherwise be broken during im-
age uploads as the uploaded image would not be immediately syncronized between all web application servers.
This will result in broken images to visitors on servers other than the node the image was uploaded on which will
persist until the media directory is resynchronized. However, as long as careful planning and execution of site
updates are followed by the site development team during new code changes and releases, the additional 11%
performance gain can be realized by using local fles.
Test 2: Apache vs. Nginx
Apache and Nginx are the two most popular HTTP servers used with Magento. Despite the performance num-
bers Nginx produces on a properly confgured web server, Apache is very competitive when also properly
confgured.
The important step to be considered is how PHP is initiated when running Apache. The standard confguration
is to use mod_php and doing so can provide reasonable performance. Apache with mod_php can be tuned to
13
perform well on moderately loaded systems but for heavily loaded systems Apache with PHP-FPM is preferred.
More importantly, however, Apache with PHP-FPM provides for full user separation that isnt easily attainable
using Apache with mod_php. For this reason we dont recommend its use in high performance environments
and do not explicitly entertain it in this whitepaper. We fnd most comparisons of Nginx and Apache are unbal-
anced because of this, as Nginx is benchmarked using PHP-FPM while Apache uses mod_php, suPHP or some
other non PHP-FPM method of handing of PHP scripts to the PHP interpreter. We benchmarked both Apache
and Nginx with PHP-FPM. This keeps the comparison balanced.
Test 2: Results
T
r
a
n
s
a
c
t
o
n
s

p
e
r

S
e
c
o
n
d
T=1 T=10 T=20 T=30 T=40 T=50 T=60 T=70 T=80 T=90 T=100
0
100
200
300
400
500
600
Apache + PHP-FPM vs. Nginx + PHP-FPM
Apache + PHP-FPM Nginx + PHP-FPM
Time in Minutes
Diagram 3: Apache + PHP-FPM vs. Nginx + PHP-FPM
The frst test run was a direct comparison of Apache with PHP-FPM and Nginx with PHP-FPM. For this Siege test
we used 250 concurrent users.
The graph shown in Diagram 3 shows the diferences between Apache and Nginx when both are utilizing PHP-
FPM. The diferences in performance between the two separate confgurations was negligible. Nginx had less
than a 1% advantage over Apache in performance over every test run. The initial ramp-up period of the graph
shows a slower transactions per second count as the various Magento caches run through their warming period.
Once the caches are fully warmed (at about 30 minutes into the test) site performance remains constant. The
variance between runs also becomes very consistent once the caches are warmed (as shown in the graphs in
Diagram 4 and Diagram 5).
14
T
r
a
n
s
a
c
t
o
n
s

p
e
r

S
e
c
o
n
d
1st minute 2nd thru 5th minute 6th thru 10th minute 11h thru 20th minute 21st thru 30th minute 31st thru 40th minute 41st thru 60th minute
Average Trans/s 95.08 114.43 152.46 273.23 485.08 547.84 555.11
+- Margin 0.00 14.24 20.45 46.33 33.34 1.94 1.61
Lower Limit 95.00 100.19 132.01 226.91 451.73 545.90 553.50
Upper Limit 100.00 128.67 172.91 319.56 518.42 549.79 556.72
0
100
200
300
400
500
600
Nginx + PHP-FPM Performance
Average Trans/s +- Margin Lower Limit Upper Limit
Digram 4: Nginx + PHP-FPM Performance
T
r
a
n
s
a
c
t
o
n
s

p
e
r

S
e
c
o
n
d
1st minute 2nd thru 5th minute 6th thru 10th minute 11h thru 20th minute 21st thru 30th minute 31st thru 40th minute 41st thru 60th minute
Average Trans/s 96.34 115.12 154.53 274.87 482.03 542.34 546.87
+- Margin 0.00 13.63 21.90 44.48 31.57 5.39 1.15
Lower Limit 90.00 101.49 132.63 230.39 450.46 536.95 545.72
Upper Limit 100.00 128.75 176.43 319.35 513.60 547.72 548.02
0
100
200
300
400
500
600
Apache + PHP-FPM Performance
Average Trans/s +- Margin Lower Limit Upper Limit
Diagram 5: Apache + PHP-FPM Performance
Choosing whether to run Magento on Apache or Nginx becomes a personal choice for those running the site
rather than one based on pure performance. One advantage of Nginx is it has a built-in proxy cache similar to
Varnish. That cache can be used to signifcantly increase performance beyond the numbers shown here, but re-
quires extensive confguration and development to make it compatible with a Magento site. This advantage over
Apache is eliminated given one can use Varnish with Apache to achieve the same results.
15
The biggest Apache advantage is the beneft of being fully compatible with Magento with no modifcations to
site rewrites or .htaccess fles needed.
Test 3: MySQL versus Percona
MySQL is an extremely popular open source relational database management system (RDBMS) for use in many
web-based applications. Magento fully supports MySQL for its database and it is the recommended database to
use in both the community and enterprise editions.
Percona is also an RDBMS, but is a drop-in replacement for MySQL and designed to ofer higher performance
over MySQL. Percona specifcally excels at InnoDB storage engine performance with its own InnoDB engine,
XtraDB. Percona also ofers built-in support for better handling of NUMA-based hardware. This is particularly
important when using very large Magento databases
The purpose of this comparison was to see if a properly tuned installation of Percona can outperform a properly
tuned instance of MySQL for a Magento site.
Test 3: Results
For the MySQL and Percona comparison, the Gatling tool was used for all testing. Running a simple Siege test
showed no diference between the two systems. Siege in not useful in this case as most data is located within
the various caches and, once warmed, the database subsystem is barely utilized. Utilizing Gatlings capabilities
to simulate actual users on the site, we were able to signifcantly stress the database and exceed more than
7,000 queries per second. The Gatling tests simulate the actual ordering process, creating a signifcant amount of
write-bound queries (which do not occur when a user simply browses the site).
The Gatling testing was performed in two parts. The frst was using various concurrencies of buyers and visitors
at a 100:1 ratio. This test was run against the MCE-SIP-200 cluster running frst with MySQL and then again with
Percona. Each test was run fve times and then averaged to compare the number of total transactions per second
against the cluster.
16
0
200
400
600
800
1000
1200
1400
1600
1800
2000
1 Buyer
100 Browsers
10 Buyers
1000 Browsers
20 Buyers
2000 Browsers
30 Buyers
3000 Browsers
40 Buyers
4000 Browsers
50 Buyers
5000 Browsers
MySQL vs. Percona
MySQL Percona
d
n
o
c
e
S

r
e
P

s
n
o
i
t
c
a
s
n
a
r
T
Diagram 6: MySQL vs. Percona
The results depicted in Diagram 6 show, as the number of buyers and visitors increase, MySQL and Percona have
fairly equal results (with a slight margin to Percona). This trend continues until the workload increased to 30
buyers and 3,000 visitors. This is where Perconas advantage became apparent. It continued to hold true un-
der an increased workload to 50 buyers and 5,000 visitors. Also, these results show all static assets returned by
Gatling mixed with orders being processed and provide a more realistic real world scenario of buyers and visitors
navigating the site.
The second part of this test was run to calculate the maximum number of orders per second an MCE-SIP-200
cluster running Percona is capable of processing. Using Gatling, a buyer test without any simulated visitors was
run. Various amounts of buyers, from 750 to 3,000, were queued and run against the cluster to see how many
orders by these buyers could be processed per second. A buyer in this test consists of the following sequence:
1. Navigate to the home page.
2. Navigate to a category page.
3. Navigate to a product page
4. Add product to the cart.
5. View the cart.
6. Proceed through checkout process fully using the check/money-order payment method.
Each calculated order is one user performing all of the above tasks successfully.
17
Percona
0
2
4
6
8
10
12
14
16
18
20
750 1000 2000 3000
O
r
d
e
r
s

P
e
r

S
e
c
o
n
d
Number of Buyers
Orders per Second Using Percona
Diagram 7: Orders per Second Using Percona
At the peak of 3,000 buyers, Magento is capable of processing 18.9 orders per second which equates to roughly
1.6 million orders per day.
Optimized Magento Site versus Optimized Magento Site with Varnish & Turpentine
Varnish is a reverse proxy HTTP application accelerator. Out of the box, Magento is not directly compatible with
Varnish due to the dynamic nature of a Magento session-based site. By default, Varnish doesnt cache requests
with cookies and Magento sends a front-end cookie with every request causing a (near) zero hit-rate for Varnishs
cache. Turpentine provides Varnish confguration fles (VCLs) to work with Magento and modifes Magentos
behavior to signifcantly improve the cache hit rate.
A test was performed utilizing Varnish along with the Turpentine plug-in and compared against a site not using
Varnish. Because user actions (such as adding items to a cart and checking out) cannot be cached, very few per-
formance diferences were noticed during these tests. Where Varnish helps is during a typical user browsing the
site.
Test 4: Best confguration versus Varnish & Turpentine
This test was performed to demonstrate the capability of a compatible and properly confgured reverse proxy in
front of Magento. For this test we utilized Varnish along with the Nexcess Turpentine plug-in on the MCE-SIP-200
cluster. Varnish was installed on each individual web application server in front of Apache and PHP-FPM.
Turpentine does not work when the full page cache is enabled due to the way sessions are handled. On a FPC
cache hit, Magento runs very little PHP code and instead pulls the cached page from disk (or memcached, Redis,
etc). With Turpentine this is a problem because it normally adds some headers to tell Varnish what is and is not
OK to cache, and whether Varnish should do ESI processing on the response (for ESI included blocks). This results
18
in the FPC caching a page with ESI injected blocks, but without varnish running the ESI processing on the page
which efectively means those blocks will be missing. Additionally, it can result in Varnish caching pages like
the customer login page that should not be cached. Thus, the FPC was disabled during the following tests with
Varnish & Turpentine. Note Varnish is so efcient at serving cached pages directly from memory, these tests were
limited by the benchmarking utility, the available network bandwidth, and benchmarking server specifcations.
Two diferent Siege tests were run with concurrency levels of 250 and 500 on each confguration.
0
1000
2000
3000
4000
5000
6000
7000
8000
250 500
d
n
o
c
e
S

r
e
P

s
n
o
i
t
c
a
s
n
a
r
T
Concurrency
Varnish / Turpentine vs. FPC
Apache / PHP-FPM + FPC Apache / PHP-FPM + Varnish / Turpentine
Diagram 8: Varnish / Turpentine vs. FPC
As can be seen above, running Varnish with Turpentine in front of Apache + PHP-FPM can provide a signifcant
performance increase. At 250 concurrent users, transactions per second increased from an average of 548 per
second to 3,800 per second with Varnish & Turpentine. At 500 concurrent users, Varnish & Turpentine again out-
performed the existing Apache + PHP-FPM setup with an average of 6,940 transactions per second over a one
hour run. This is a benchmarked 1,300% increase in performance, which potentially provides over 600 million
transactions per day.
19
1
10
100
1000
10000
T
=
1
T
=
2
T
=
3
T
=
4
T
=
5
T
=
6
T
=
7
T
=
8
T
=
9
T
=
1
0
T
=
1
1
T
=
1
2
T
=
1
3
T
=
1
4
T
=
1
5
T
=
1
6
T
=
1
7
T
=
1
8
T
=
1
9
T
=
2
0
T
=
2
1
T
=
2
2
T
=
2
3
T
=
2
4
T
=
2
5
T
=
2
6
T
=
2
7
T
=
2
8
T
=
2
9
T
=
3
0
T
=
3
1
T
=
3
2
T
=
3
3
T
=
3
4
T
=
3
5
T
=
3
6
T
=
3
7
T
=
3
8
T
=
3
9
T
=
4
0
T
=
4
1
T
=
4
2
T
=
4
3
T
=
4
4
T
=
4
5
T
=
4
6
T
=
4
7
T
=
4
8
T
=
4
9
T
=
5
0
T
=
5
1
T
=
5
2
T
=
5
3
T
=
5
4
T
=
5
5
T
=
5
6
T
=
5
7
T
=
5
8
T
=
5
9
T
=
6
0
T
r
a
n
s
a
c
t
i
o
n
s

P
e
r

S
e
c
o
n
d
Time (minutes)
Varnish / Turpentine vs. FPC
Apache / PHP-FPM + Varnish / Turpentine Apache / PHP-FPM + FPC
Diagram 9: Varnish / Turpentine vs. FPC
Looking at the transaction rate graph above shown with a logarithmic scale, the non-Varnish & Turpentine
confguration shows the ramp-up period of the full page cache over the frst 30 minutes. With Varnish, the cache
is immediately populated in the frst 60 seconds of the run since nearly every single page has been hit during
this time frame. This high rate of transactions per second is maintained over the 60 minute test with some
observed variance. The reason behind this variance comes down the capabilities of the benchmarking server. At
this rate our benchmarking server was seeing transfer rates of over 900Mb/s to the MCE-SIP-200 cluster which
was ultimately limiting our results. Running the Gatling test with concurrencies higher than 3,000 provided no
additional performance numbers as the gigabit connection between the cluster and the benchmark server were
saturated.
Running the Gatling buyers-only test to calculate orders per second with Varnish & Turpentine resulted in only
a moderate performance increase to 21.2 orders per second (up from 18.5). This is expected as most steps in the
buying process are HTTP POST operations and therefore cannot be cached.
20
Result Observations and Optimal Confgurations
The Importance of the Magento Enterprise Full Page Cache
One of the biggest built-in performance improvements in Magento is the full page cache. When enabled, and
once fully warmed, the number of transactions per second was over 500% higher compared to the same site
with the FPC disabled (see Diagram 10).
T
r
a
n
s
a
c
t
i
o
n
s

P
e
r

S
e
c
o
n
d
0
100
200
300
400
500
600
T
=
1
T
=
3
T
=
5
T
=
7
T
=
9
T
=
1
1
T
=
1
3
T
=
1
5
T
=
1
7
T
=
1
9
T
=
2
1
T
=
2
3
T
=
2
5
T
=
2
7
T
=
2
9
T
=
3
1
T
=
3
3
T
=
3
5
T
=
3
7
T
=
3
9
T
=
4
1
T
=
4
3
T
=
4
5
T
=
4
7
T
=
4
9
T
=
5
1
T
=
5
3
T
=
5
5
T
=
5
7
T
=
5
9
T
=
6
1
T
=
6
3
T
=
6
5
T
=
6
7
T
=
6
9
T
=
7
1
T
=
7
3
T
=
7
5
T
=
7
7
T
=
7
9
T
=
8
1
T
=
8
3
T
=
8
5
T
=
8
7
T
=
8
9
T
=
9
1
T
=
9
3
T
=
9
5
T
=
9
7
T
=
9
9
Full Page Cache Enabled vs. Disabled
FPC Disabled FPC Enabled
Time In Minutes
Diagram 10: Full Page Cache Enabled vs. Disabled
The graph presented in Diagram 11 shows the relationship of the fast backend, slow backend, and full page
cache utilization in megabytes along with the hit ratios for all three caches (in percentages). The number of
benchmarked transactions per second is graphed alongside these cache variables to correlate the cache usage
versus site performance.
21
C
a
c
h
e

H
i
t

P
e
r
c
e
n
t
a
g
e

0
10
20
30
40
50
60
70
80
90
100
0
500
1000
1500
2000
2500
3000
3500
4000
4500
T
=
1
T
=
3
T
=
5
T
=
7
T
=
9
T
=
1
1
T
=
1
3
T
=
1
5
T
=
1
7
T
=
1
9
T
=
2
1
T
=
2
3
T
=
2
5
T
=
2
7
T
=
2
9
T
=
3
1
T
=
3
3
T
=
3
5
T
=
3
7
T
=
3
9
T
=
4
1
T
=
4
3
T
=
4
5
T
=
4
7
T
=
4
9
T
=
5
1
T
=
5
3
T
=
5
5
T
=
5
7
T
=
5
9
T
=
6
1
T
=
6
3
T
=
6
5
T
=
6
7
T
=
6
9
T
=
7
1
T
=
7
3
T
=
7
5
T
=
7
7
T
=
7
9
T
=
8
1
T
=
8
3
T
=
8
5
T
=
8
7
T
=
8
9
T
=
9
1
T
=
9
3
T
=
9
5
T
=
9
7
T
=
9
9
Cache Utilization vs. Transactions Per Second
Full Page Cache Usage (MB)
Fast Cache Usage (MB)
Slow Cache Usage (MB)
Transfer Rate (Transactions Per Second)
Full Page Cache Hit Rate Percentage
Fast Cache Hit Rate Percentage
Slow Cache Hit Rate Percentage
Time In Minutes
M
B

u
s
e
d

/

T
r
a
n
s
a
c
t
i
o
n
s

p
e
r

S
e
c
o
n
d

Diagram 11: Cache Utilization vs. Transactions Per Second
The resulting data shows the sites transactions per second performance is heavily dependent upon the caches
being fully populated with data. All three caches ramped-up in the frst 30 minutes of benchmarking and then
remained at a constant level over time, thereby providing the best performance. Because the hit rate is averaged
over time, once the frst minute of data was available, the hit rate was already over 75% for both the fast cache
and full page cache. It then continued to climb upwards as the test progresses. Also note the data in the fast
cache was fully populated and the fast cache was large enough to hold all data necessary, thus the slow cache
was rarely ever used (even though it was also fully populated).
The Magento DEFAULT_LIFETIME
During some three hour-long and higher benchmark runs, an interesting artifact was observed. Roughly two
hours into the benchmark, a slight dip would occur, deviating from the standard benchmark baseline estab-
lished once all caches were built.
22
0
2
4
6
8
10
12
14
T
=
1
T
=
4
T
=
7
T
=
1
0
T
=
1
3
T
=
1
6
T
=
1
9
T
=
2
2
T
=
2
5
T
=
2
8
T
=
3
1
T
=
3
4
T
=
3
7
T
=
4
0
T
=
4
3
T
=
4
6
T
=
4
9
T
=
5
2
T
=
5
5
T
=
5
8
T
=
6
1
T
=
6
4
T
=
6
7
T
=
7
0
T
=
7
3
T
=
7
6
T
=
7
9
T
=
8
2
T
=
8
5
T
=
8
8
T
=
9
1
T
=
9
4
T
=
9
7
T
=
1
0
0
T
=
1
0
3
T
=
1
0
6
T
=
1
0
9
T
=
1
1
2
T
=
1
1
5
T
=
1
1
8
T
=
1
2
1
T
=
1
2
4
T
=
1
2
7
T
=
1
3
0
T
=
1
3
3
T
=
1
3
6
T
=
1
3
9
T
=
1
4
2
T
=
1
4
5
T
=
1
4
8
T
=
1
5
1
T
=
1
5
4
T
=
1
5
7
T
=
1
6
0
T
=
1
6
3
T
=
1
6
6
T
=
1
6
9
T
=
1
7
2
T
=
1
7
5
T
=
1
7
8
M
B
p
s
Website Throughput
Time in Minutes
Diagram 12: Website Throughput
The cause of this dip results from the fact there is a default lifetime variable in
app/code/core/Mage/Core/Model/Cache.php set to 7,200 seconds (or two hours).
class Mage _ Core _ Model _ Cache
{
const DEFAULT _ LIFETIME = 7200;
const OPTIONS _ CACHE _ ID = core _ cache _ options;
const INVALIDATED _ TYPES = core _ cache _ invalidate;
const XML _ PATH _ TYPES = global/cache/types;
At the 7,200 second expiry time, data will begin to be automatically purged from the various caches. This data
is immediately repopulated into each cache, keeping each caches hit rate high, but the purging is enough to
cause a 16MB/s drop in the outgoing trafc along with an 80 transaction per second drop at peak fushing.
23
Time in Minutes
T
r
a
n
s
a
c
t
i
o
n
s

p
e
r

S
e
c
o
n
d
0
100
200
300
400
500
600
T
=
1
T
=
4
T
=
7
T
=
1
0
T
=
1
3
T
=
1
6
T
=
1
9
T
=
2
2
T
=
2
5
T
=
2
8
T
=
3
1
T
=
3
4
T
=
3
7
T
=
4
0
T
=
4
3
T
=
4
6
T
=
4
9
T
=
5
2
T
=
5
5
T
=
5
8
T
=
6
1
T
=
6
4
T
=
6
7
T
=
7
0
T
=
7
3
T
=
7
6
T
=
7
9
T
=
8
2
T
=
8
5
T
=
8
8
T
=
9
1
T
=
9
4
T
=
9
7
T
=
1
0
0
T
=
1
0
3
T
=
1
0
6
T
=
1
0
9
T
=
1
1
2
T
=
1
1
5
T
=
1
1
8
T
=
1
2
1
T
=
1
2
4
T
=
1
2
7
T
=
1
3
0
T
=
1
3
3
T
=
1
3
6
T
=
1
3
9
T
=
1
4
2
T
=
1
4
5
T
=
1
4
8
T
=
1
5
1
T
=
1
5
4
T
=
1
5
7
T
=
1
6
0
T
=
1
6
3
T
=
1
6
6
T
=
1
6
9
T
=
1
7
2
T
=
1
7
5
T
=
1
7
8
Website Transactions
Diagram 13: Website Transactions
Since the cache was pre-warmed before the test runs all of the caches expired around the same time. On a lower
trafc site, the expiration of cached pages will be more graceful and would not cause such a pronounced dip as
seen here. Bear in mind, on a high trafc site, fushing the cache will cause a fast cache build and similar expi-
ration behavior as seen in Diagram 13. Because of this, it is not recommended to fush any caches during peak
trafc hours.
Response Times
An important part of any Magento store is fast page response time. Benchmarking the MCE-SIP-200 cluster with
a 250 concurrent user Siege test and the full page cache enabled, page response times were sustained at under
a half second once the cache was warmed roughly 30 minutes into the test. The exact same benchmark ran with
the full page cache disabled in Magento Enterprise resulted in a best page response time of 2.04 seconds at 250
concurrent connections. This again reinforces the importance of the FPC within Magento Enterprise.
24
0
0.5
1
1.5
2
2.5
3
3.5
T
=
1
T
=
3
T
=
5
T
=
7
T
=
9
T
=
1
1
T
=
1
3
T
=
1
5
T
=
1
7
T
=
1
9
T
=
2
1
T
=
2
3
T
=
2
5
T
=
2
7
T
=
2
9
T
=
3
1
T
=
3
3
T
=
3
5
T
=
3
7
T
=
3
9
T
=
4
1
T
=
4
3
T
=
4
5
T
=
4
7
T
=
4
9
T
=
5
1
T
=
5
3
T
=
5
5
T
=
5
7
T
=
5
9
T
=
6
1
T
=
6
3
T
=
6
5
T
=
6
7
T
=
6
9
T
=
7
1
T
=
7
3
T
=
7
5
T
=
7
7
T
=
7
9
T
=
8
1
T
=
8
3
T
=
8
5
T
=
8
7
T
=
8
9
T
=
9
1
T
=
9
3
T
=
9
5
T
=
9
7
T
=
9
9
Response Time
Without FPC With FPC
Time In Minutes
P
a
g
e

L
o
a
d

T
i
m
e

I
n

S
e
c
o
n
d
s
Diagram 14: Response Time
25
Understanding Magento Caching
When Magento and cache are used in the same sentence, many more details must be provided to be sure
any caching mechanism being talked about is understood. There are many diferent caches that can be used in
(and alongside) Magento, including, but not limited to, a PHP opcode cache, session cache, full-page cache, fast
backend cache, slow backend cache and Varnish cache. Properly implementing and confguring these can result
in considerable performance increases compared to a site without caching or without a correctly confgured
caching system.
Magento Backend Cache
The Magento backend cache is the most common cache used as it is a native part of the Zend Framework (of
which Magento takes advantage). This cache can be confgured in two diferent implementations: a single
backend or a two-level backend cache. There are also many methods of storage the Magento backend cache
can be confgured to use, including the local fle system, memcached, Redis and APC. When enabled, the default
method is to use the local fle system to store cache data as individual fles on disk. While this is an improvement
over not using caching at all, there are better memory-based options available such as memcached or Redis that
can greatly enhance performance.
Redis is preferred because of its tag support (something the other memory cache engines lack). When Magento
was confgured to use Redis as the sole caching backend in a single, non-two-level confguration, a slowdown
in transactions per second was noticed while performing our benchmark tests. Upon further investigation, it
was realized Redis, being a single-threaded application, does not take advantage of multi-core CPU systems.
The lack of thread support, along with the high number of incoming serialized requests, had a slight efect on
performance. Redis documentation acknowledges this limitation and suggests using parallel Redis instances in
situations like this. Unfortunately, as of this writing, these separate instances are not truly distributed and fault
tolerant. Redis is aware of this and is working on a product named Redis Sentinel which should solve this issue.
We will revisit this testing with the latest stable version of Redis as a single-level cache solution when the tools
permit.
A two-level cache confguration was also tested and resulted in the highest benchmarked transaction numbers
and best site functionality. Redis was used as the tag-supporting slow, or second level, cache, and memcached
as the frst level or fast cache. The beneft of this setup was we could use the multi-threaded capabilities of mem-
cached while retaining the tag-support of Redis (all without sacrifcing memory-backed data stores for disk I/O).
A second dedicated Redis instance was also set up for exclusive use by the full page cache module in Magento
Enterprise (detailed below).
One last thought should be kept in mind regarding the backend cache and APC. APC is an allowed cache meth-
od to be used throughout the various Magento cache confgurations, but its use is not recommended. APC is
tied to the running PHP-FPM parent process, and, in the event PHP-FPM is restarted, it will forcefully fush your
cache (which is not an ideal behavior). If APC is also being used for a PHP opcode cache, using it for the backend
cache will result in greater APC memory fragmentation.
26
Additional improvements
When using Magentos two-level cache built on the Zend Framework, there is a known bug preventing the
cache from operating correctly. The result of this bug prevents the frst level cache from refreshing correctly,
which in turn makes the frst level cache perform much worse than it should. There is a patch available that
will fx this bug and it is recommended for use in all two-level cache implementations. Information on this bug
and a patch can be found at http://www.fabrizio-branca.de/magento-zend-frameworks-twolevels-cache-back-
end-mess.html. All benchmarks in this white paper were run with the Zend two-level cache bug fx implement-
ed.
Magento Full Page Cache
The full page cache is a feature ofered in Magento Enterprise. The basic idea of the Magento full page cache is
to store full HTML output of a page request. It is another cache which can be set to use various backend mech-
anisms such as fles, or memory-based using memcached or Redis. When running Magento with the full page
cache enabled, and having the cache be fully populated and hot, a signifcant performance increase can be
observed.
Magento Session Cache
Sessions within Magento are stored within their own cache and the location of this cache is customizable. The
fle system is the default location for sessions to be stored, and while this is fne for most purposes, additional
performance can be gained by removing the fle system from the session-based operations and storing sessions
in memory via memcached. Using the database for session storage is another option, although it is not normally
recommended.
It is important to be sure the session storage location is shared between all web application servers in a
multi-server setup. Load balancer persistence (if enabled) will keep connections on the same web server, but in
the event that the client moves to an additional server, or switches HTTP to HTTPS, having a centralized session
location will ensure the clients credentials and current cart remain consistent.
It is also a good practice to create an additional memcached instance on a diferent host or port to store session
data. Using the same memcached instance for both Magento backend cached data and session data is not rec-
ommended, as fushing the Magento backend cache will also fush the clients session data.
PHP opcode cache
There are many Magento-compatible open-source PHP opcode optimizers and caches. These caches (such as
APC, eAccelerator, and Xcache) will cache the bytecode compiled from source by PHP. The advantage of this is
frequently used pieces of code can be cached, thereby saving processing time as PHP does not have to recom-
pile the same piece of code multiple times. While Magento will work with APC, eAccelerator, and Xcache, APC
is the recommended opcode cache to be used with Magento. While all three caches ofer similar performance,
only APC is used in all benchmarking tests. For all tests in this white paper, APC was enabled on both web appli-
cation servers.
27
Conclusion and Additional Performance Recommen-
dations
As demonstrated by the above testing and benchmarking procedures, a load-balanced, two web application
server cluster with a single database server can process 47 million transactions and over 1.6 million orders per
day. Thus, the following should be considered best practices for a Magento installation for maximum perfor-
mance:
Run either Nginx or Apache with PHP-FPM. Both ofer nearly identical performance utilizing PHP-FPM,
but Apache benefts include better drop-in compatibility with Magento than Nginx. While Nginx re-
quires more custom confguration to work with Magento, it does ofer built-in reverse proxy capability.
Bear in mind, one does not simply turn this reverse proxy on; it will require extensive confguration to
work with a production ready Magento store.
Enable the full page cache when using Magento Enterprise with its own dedicated Redis backend. The
FPC can ofer a signifcant increase in performance over a Magento site not utilizing it. Avoid fushing
the FPC during peak trafc hours as it can severely afect site performance while the cache rebuilds.
Please note that sharing the FPC Redis cache instance with the Magento 2nd level cache (or any other
caching module) is not recommended and can actually degrade performance. Make sure the FPC Redis
instance is dedicated to the FPC.
Utilize a two-level cache with memcached as the fast cache and Redis as the slow cache. Be sure avail-
able memory in each of these caches is high enough to prevent forced evictions. Evictions can be
checked by observing the built-in stats for each cache instance. If using the two-level cache, be sure to
apply the two-level cache bug fx.
Use a dedicated memcached instance for sessions. Be sure this instance is shared across all web applica-
tion nodes so session data is available to a user on any server node and protocol.
Use local fle systems over NFS. Local fle systems will ofer an 11% performance increase over NFS
mounted fle systems at the cost of additional complexity. If your actual trafc is well under the maxi-
mum trafc capabilities of your cluster, you can use NFS for overall simplicity of site maintenance and
operation.
Use Percona over MySQL if Magento will be used in high trafc installations. Percona provides better all
around performance with Magento along with consistent write performance.
Use Varnish as a reverse proxy server in front of your HTTP server. Utilize the Nexcess Turpentine plug-in
for better compatibility between Varnish and Magento. In this confguration, a transaction per second
increase of 1,300% beyond the 47 million transactions per day cited, along with a 15% improvement in
orders per second can be realized.
Bear in mind, all results here were tested on Magento Enterprise without any optional extensions or plug-ins in-
stalled with exception of the Cm_RedisSession add-on for Redis functionality and the Nexcess Turpentine plugin
where marked as used. The addition of Magento extensions and plug-ins, depending on their functionality, inte-
28
gration, and usage, can drastically afect overall site performance. It is always a good practice to benchmark your
site before and after an extension installation so you are aware of any performance degradation from it.
Appendix A: System Confgurations
All system level confguration fles used for benchmarking, including the Magento Enterprise local.xml confgu-
rations, are available on github at: https://github.com/nexcess/whitepaper-v1-confgs.
Appendix B: Benchmarking Confgurations
Gatling Tool
All code used for the Gatling Tool tests is available on github at: https://github.com/nexcess/magento-gat-
ling-tests
Siege
The following command line option was used for all Siege tests:
/usr/local/bin/siege -c $CONCURRENCY -i -b -t60s -f urls.txt --log=$SIEGELOG -A
Siege
urls.txt is the Magento sitemap parsed into a text fle. $CONCURRENCY is the concurrency passed to the test.
$SIEGELOG is the path to log the output from Siege.

Magento Performance Testing For High Performance

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Magento Performance Testing For High Performance

Uploaded by

Copyright:

Available Formats

MAGENTO HOSTING

Best Practices for Optimum Performance v1.6

You might also like