The Guarantee
July 6

Marketer’s Secret Weapon – Multivariate Testing

The market is likely close to hitting the bottom — with a fairly quick ascension anticipated in many verticals over the next couple quarters — and most communities are in the middle of a phased return to the workplace. While we may never see a return to business as usual, we are on the path to a “new normal” which is already serving as a signal to business executives to ramp up their marketing.

With the noise out there competing for the attention of your prospects already, the volume of which is anticipated only to increase over the coming two months, how do you ensure that your message will actually break through? What’s more, how do you ensure your marketing messages are appropriate — and not tone deaf — given all that’s going on in the world? The answer to both questions is “split testing” on a grand scale … a marketer’s secret weapon.

Leveraging Split Testing

Split testing, also referred to as A/B testing or multivariate testing, is a method of conducting a series of controlled, randomized experiments with the goal of selecting the strongest campaign and fully optimizing its effectiveness. The objective for a traditional split test is to wait for a statistically significant difference in consumer behavior to occur as a result of seeing one ad versus another — with the only difference between the two being a single variable such as the image used or headline. You then continue to test additional variables within that ad over time until it’s fully optimized. The key to an effective split test is ensuring that only a single variable changes each time. For example, you might test the same ad design with 5 different headlines. Or the same design with 5 different calls to action. The most commonly tested variables are: images and other design elements, color palette, font, copy, headlines, and calls to action.

The fundamentals of this approach to testing new advertising campaigns align well with a business philosophy of author Jim Collins — outlined in his book “Great by Choice” — where he shares the benefits of a bullets-then-cannonballs (figuratively speaking) approach to innovation testing and recalibration. Please forgive the reference to ammunition in light of today’s troubled world, but it’s an essential comparison to make the point.

Bullets Then Cannonballs?

Here’s an analogy, from Collins, that will help cement the concept in your mind. “Picture yourself at sea, a hostile ship bearing down on you. You have a limited amount of gunpowder. You take all your gunpowder and use it to fire a big cannonball. The cannonball flies out over the ocean…and misses the target, off by 40 degrees. You turn to your stockpile and discover that you’re out of gunpowder. You die. But suppose instead that when you see the ship bearing down, you take a little bit of gunpowder and fire a bullet. It misses by 40 degrees. You make another bullet and fire. It misses by 30 degrees. You make a third bullet and fire, missing by only 10 degrees. The next bullet hits — ping! —the hull of the oncoming ship. Now, you take all the remaining gunpowder and fire a big cannonball along the same line of sight, which sinks the enemy ship. You live.”

The idea of firing bullets and then cannonballs in business-innovation testing is inspired by this analogy. First, you “fire bullets (low-cost, low-risk, low-distraction experiments) to figure out what will work — calibrating your line of sight by taking small shots. Then, once you have empirical validation, you fire a cannonball (concentrating resources into a big bet) on the calibrated line of sight. Calibrated cannonballs correlate with outsized results; uncalibrated cannonballs correlate with disaster. The ability to turn small, proven ideas (bullets) into huge hits (cannonballs) counts more than the sheer amount of pure innovation.”

Simple A/B Split Testing Vs. Aggressive Multivariate Testing

So let’s talk specifically and practically about how to implement a bullets-then-cannonballs approach to your return-to-market advertising campaigns to ensure they are appropriate and breaking through.

You may elect to simply run one test at a time and continue to iterate your ad campaign until further refinements offer diminishing returns. Note that you’ll need to test your campaign separately through each channel, as certain images will perform better on social media versus email, for example.

While testing just one variable at a time can be a painstaking and lengthy process, there is another way — multivariate testing — where you are testing a variety of elements all at once. This is the fastest path to clarity around messages that are appropriate for our times and most likely to elicit the desired behavior (e.g., clicking or buying).

Here’s a sample map of what a multivariate test of a Facebook ad campaign could look like if you’re testing three images, three ad titles, and five precise prospect interests for fine targeting.  These prospect interests, by the way, are determined by Facebook based on the pages the audience likes, the posts they engage with, etc. As you can see from the graph that follows, this results in testing 45 different ad scenarios.


Manually creating 9 distinct ads – each of which Facebook will test across 5 different prospect interest types – is an arduous process for a single campaign. Thankfully there are split-testing platforms that can automate much of this. It’s critical that your split testing occur to a large enough audience to provide you with statistically valid results, and most reputable split-testing platforms will ensure that’s the case.

Exploring Partnership

If you are looking for a partner to develop your return-to-market ad campaigns and implement split testing to ensure you don’t fire that cannonball too soon, we’d be delighted to speak with you.

As long as COVID-19 is decimating our economy and business outcomes, each week I’ll be sharing the most effective growth strategies, like these, that forward-thinking companies are executing during this crisis. My hope is that you’ll find additional inspiration to advance your vision forward with a full-on strategic offensive, and that includes the sales and marketing front.

We are all in this together. RedRover is deeply committed to businesses fighting their way through the current health and economic crisis. Let us know if you’d like a hand retooling your sales and marketing strategies in order to ensure growth during and after this crisis. Knowing what to do next can be difficult, especially when you are so close to your business, and we stand ready to help.

Lori Turner-Wilson is founder and CEO of RedRover Sales & Marketing Strategy. A fast-growing agency of seasoned professionals, RedRover is one of the few agencies in the country to integrate sales training with marketing strategic planning and execution. RedRover has a uniquely intense focus on achieving measurable results for its clientele, as one of only a few agencies across the country offering its clients a results guarantee. The agency’s diverse client roster represents nearly every industry vertical — both B2B and B2C.


Fire Bullets, Then Cannonballs,
Facebook Ads Split Testing in 2019,

Related Posts