Which A/B testing tools or multivariate testing software should

you use for split-testing?

blog.getscenario.com
blog.getscenario.com
blog.getscenario.com
 COLUMN TITLES

A/B OR MVT? - In an A/B test, your visitor sees Version A of a page or Version B of it. (Note, a test with three page versions would be called
an A/B/C test. “A/B/n” is often used as shorthand for a test that has n different page versions.) Multivariate (MVT) tests are simply ones in
which you carry out A/B tests (or A/B/n ones) on several independent page elements simultaneously.

TYPE OF MVT - There are many different ways of carrying out multivariate (MVT) tests: You can show every single possible combination of
page elements. This is called “full factorial”. Or you can show only a fraction of them. This is called “fractional factorial”. There are loads of
types of fractional factorial test, and they all have complicated-sounding names. (Taguchi is one of them). You can also choose to display los-
ing combinations less frequently. The data can be analyzed by the software in different ways too.

CAN MULTIPLE GOALS BE SET ? - Sometimes you may want to track more than one goal. For example… (i) You may want to know the effect
that your experiment has not only on orders, but also on leads submitted. (ii) You may be concerned that you’re increasing the sale of one
product at the expense of the sales of others. In this case, you’d want to track the effect that the experiment has on the sales of each product.
(iii) You may wish to count conversions at several stages down the conversion funnel. That way, if you’re aiming to increase, say, the number
of orders, but your test doesn’t reach statistical significance, you may instead choose to consider an intermediate goal such as click-throughs
(such as add-to-cart actions), and make the assumption that the number of orders would be proportional to the number of click-throughs.

CAN TRAFFIC BE SEGMENTED ? - You may discover that different types of traffic prefer different versions of your page. For example, new
prospects and existing customers may prefer different versions of a webpage. Some software vendors recommend that their software can
act as a “traffic cop”, forever directing different types of visitor to different optimal versions of the webpage long after the test has concluded.
Though this feature may have great benefits for you, be aware that a vendor benefits greatly from being able to “lock you in” in this manner,
so they may be tempted to over-enthusiastically sell the benefits of this feature.

HAS A PLATFORM FOR CREATING WEBPAGES - Some of the software tools have in-built platforms for creating and editing webpages — a
bit like having their own WordPress interface. They can be useful if the thought of involving your company’s web development team gives you
the shudders.

blog.getscenario.com
SELF SERVICE OR FULL SERVICE - “Self service” means you just get the software (and maybe some support). “Full service” usually involves *
technical support * assistance in deciding “what to test” * designing the experimental plans * interpreting the data

TABLE BODY

CLIENT-SIDE - means the software is executed by the visitor’s browser (for example, Internet Explorer or Firefox). This is done by “tagging” the
webpage’s HTML with several lines of Javascript. When the page is loaded, the Javascript makes calls to a back-office system (such as Google’s
Analytics servers), which then notifies the browser of which page content should be downloaded and displayed. The back-office system also keeps
score of conversions.

SERVER-SIDE - means the software runs on the server. The page is compiled by the programming language on the server, and is presented to the
browser in its finished form. In the same way that some PC software is only available for Windows or for Macs, some server-side software is only
available for particular server technologies (such as PHP).

blog.getscenario.com