Why A/B Testing is a Complete Waste of Your Time

a:bMost of your A/B tests are a complete waste of time.

That’s right, they don’t do you any good.

Why? Well, picking random tests seldom leads to fantastic results.

And this comes from what KINDs of tests you’re doing.

With A/B testing and optimization, most people think of headlines, button colors, and layouts. And in the process, we completely miss a much more important kind of testing.

Are You Optimizing Locally or Globally?

You’re using a set of strategies, frameworks, and assumptions to grow your business. This is the structure of your marketing funnel, your target market, and your positioning. Each of these forces you into a set of constraints.

No matter how many colors you test on your buttons, you won’t be able to break free from these constraints. Any optimization that you do will get you closer to the local maxima. It’s the maximum results you could possibly get from the main constraints you’ve already chosen.

Even if you optimize every last piece of your marketing funnel, you’ll never be able to break through the local maxima.

Early this year, we ran countless tests on our webinars at KISSmetrics. Signup flows, calls to action, copy, offering recordings, it all got tested. Here’s a few things we learned:

  • Telling people that there WON’T be a recording of your webinar boosts attendance by 50%. You’ll get about a third of your registrants to show up.
  • Even with a short signup form, you can boost signups by breaking it into multiple steps. Ask for just an email on step 1 and then load a second version with the other fields you need (even if it’s just a name). Once people commit a little bit, they’ll be much more inclined to follow through.

Insights like these helped us find the local maxima when it came to webinars.

But there might be an even better way to get people interested in our product. Maybe webinars aren’t the most efficient method, maybe we need to revamp our positioning, or maybe we need to completely rethink how we nurture our audience. Standard A/B tests won’t ever answer these questions.

Instead, we need to look for the global maxima.

Finding the Global Maxima

function-max-global

The global maxima is the single best outcome you could possibly achieve regardless of where you’re starting from. Brand positioning, target markets, the main benefits of your product, channels, viral loops, and lead generation strategies, move you closer to the global maxima. Or they might move you further away and limit your results event more. Bigger rewards come with bigger risks.

You’ll want to be careful when pursuing global maximas.

In normal A/B testing, you can usually just follow the data. Version B won? Great! Let’s use it.

Data gets a LOT fuzzier when you’re trying to test for global maximas.

Why? A new strategy typically doesn’t perform as well in the short term as what you’re doing currently.

Let’s say you decide to test a new target market. It’s going to take some time to figure out how to reach that new group of customers before you even know whether or not it’s a viable plan. In the meantime, you’ll see a dip in revenue while you get everything up and running.

You might have a real winner in the long term but in the short term, it won’t look nearly as good as your current approach.

At the end of the day, you probably won’t find THE global maxima. But you should be looking for one that’s pretty good. Once you’ve found a good model to build a business off of, then you can drill down and optimize for the local maxima.

Changing Your Optimization Process

Core aspects of your marketing like these help you move closer to the global maxima:

  • Positioning
  • Best channels for customer acquisition
  • The structure of your marketing funnel

You’ll need to be testing major changes and giving each test the time it needs to get enough data. Come up with radical tests and look for completely different approaches. Small tweaks won’t help you here. Go big.

Once you’ve found your marketing global maxima, then start A/B testing the typical stuff like:

  • Landing pages
  • Home pages
  • Emails
  • Copy
  • Design
  • SEO

Most marketers get trapped into optimizing for the local maxima without ever testing their global maxima.

Each and every time you run an A/B test (or any test for that matter), ask yourself if you’re testing a local maxima or a global maxima. And if you’re still trying to figure out the core model for your marketing team, you’ll need to test completely different ideas in order to find that global maxima.


About the Author: Lars Lofgren is the Growth Manager at KISSmetrics and his experiences include:

  • Digital marketing consultant that has delivered incredible results like doubling the conversion rate of e-commerce sites and building sales pages with a 70% click through rates for small to large size businesses
  • Guest Lecturing at the University of Colorado at Boulder
  • Running online marketing at American Alpine Club and KISSmetrics
Tags: , , ,
Free Guide: Straight-up Startup Advice

Learn From The Best
Top Entrepreneurial Advice
150 Entrepreneurs Share Their Secrets

Free Guide: Straight-up Startup Advice

Comments.

  • jana

    It may be helpful to define local and global maxima. This is a mathematical term that I don’t quite understand. Maybe I’m missing something, but this article confused me.

    • Lars Lofgren

      The global maxima is the single best outcome you could ever achieve. But to reach it, you’d need to make your target market, product, and marketing strategy perfect.

      The local maxima is the best outcome you can achieve given your current choices.

      A/B tests will help you find the local maxima but not the global maxima.

  • ZachJex

    He defined both.

    Local maximum – the maximum results you could possibly get from the main constraints you’ve already chosen.

    Global maximum – The global maximum is the single best outcome you could possible achieve regardless of where you’re starting from.

    In basic terms…

    If you start with “which of the two buttons is better”, you’re only getting the best possible outcome between THOSE TWO. This equals local maximum.

    He’s saying that you should be chasing the global maximum – which asks what the best possible outcome is starting from ANY point (not just which button is better). Maybe the answer is not based on buttons, for instance.

    It could have been presented in a more down to earth way, but I found the content interesting.

  • acrookston

    Sensational click-bait title, and I fell for it, curious to see who is it now that doesn’t understand how to write a real A/B test. I’m glad you’re on your way to doing so…

  • Brian Crouch

    “The global maxima is the single best outcome you could possibly achieve, regardless of where you’re starting from.”
    I am hesitant to wade in here, but “possibly” could mean 100% in all instances. 100% conversion, 100% CTR, etc. Perhaps “realistically” would be a better adverb there.

    • Lars Lofgren

      The best realistic outcome is a good way to think of it. Obviously, you’ll never have 100% conversion rates, 100% re-purchase rates, or a 0% churn. Every marketing funnel leaks at least a little bit.

  • Samuel Hulick

    “Telling people that there WON’T be a recording of your webinar boosts attendance by 50%.”

    How are you arriving at this number? The only way I can think to control for variables like webinar topic, seasonality, etc. is to include that message in some invitations and not others for the same webinar and then compare their conversion rates. However, that would be an A/B test and, per your definition, a complete waste of time.

    • Lars Lofgren

      We tested it over a series of webinars instead of running an A/B test on a single one. Then I pulled conversion rates on registrants -> attendees. Whenever recordings were offered, attendance dropped by the same percentage every time.

      • Samuel Hulick

        Then it sounds like you could have SAVED time with an A/B test by not having to add and remove it from multiple webinars. Plus, you could have enjoyed the higher conversion rate on all the ones you pulled it from after the first.

        • Lars Lofgren

          A single A/B test doesn’t provide conclusive data. Customer behavior changes, other sources of traffic behave differently, and seasonality is always shifting outcomes. Running tests under a number of different conditions and getting the same results gives you more robust data.

  • anatolyg

    This article says to me, “think of crazier changes, make those your B, and then do an A/B test.” The only difference is the time you give a test, and the types of bold changes you’re willing to make.

  • Yuriy Sklyar

    Sounds like this article compares 2 types of testing: multivariate (smaller changes like buttons, copy, ect) vs actual A/B testing (major layout/graphic changes)…

  • shadyyx

    I think reading of this article is also a waste of time (I saved some by not reading it all to write this comment).

    I always thought that A/B testing is not about finding any maximum but about deciding what UI/UX is better for the most of users right now – because there is no maximum at all – tomorrow a new technology may be born and a new (global) maximum could be achieved.

    And I also always thought that only those who have plenty of resources, plenty of time and plenty of money can afford the A/B testing so this article didn’t bring nothing new to the table.

    And as somebody already said here: “Sensational click-bait title” – with almost nothing to the theme inside the bubble.

  • Grigoriy Kogan

    This is pure link-bait: Sensational title + Unoriginal article. I’m unsubscribing from this blog.

  • Prashant Chambakara

    Let me say from my end, I totally disagree with your view that it’s completely a waste if time. When it comes to optimizing a website, we often get confused on either to consider AB testing or Multivariate testing. I believe both have their own significance. I’ve shared an article on their importance, please check it out here http://tech.pro/blog/1710/ab-and-multivariate-testing–a-quick-go-through

  • Andreea

    Which shows that is not A/B testing who’s increasing conversions, it’s actually the way you do A/B tests. It’s all about the right moment, having enough data and traffic, having the right tools. We’ve been recently using Marketizator, and it seems ok.

  • Alison Lyne

    The main idea here to “think big” when testing is fundamentally good advice. However, I think the author’s point gets a bit lost with the warning against “picking random tests” and recommendation instead to “come up with radical tests.” …Be strategically random-er, but how?

    From the comments, It looks like people were expecting more of a clear strategy, or system for reaching the Global Maximum. Persado does this = With a platform based in math & big data to canvas the space of possible messaging (emotions, product features, ect) and systematically improve marketing language. Optimizely also gives great advice in their book on A/B Testing, I’ve attached a snippet on this subject. It’s a great read!