I’ve met a lot of amazing marketers at startups and smaller companies in the last month and a half who are just beginning to get their feet wet with testing. Most of them have run 1 or 2 tests, and are hesitant to take the next step of investing more time & money in resources and/or a platform. Here’s the question I get most often: is testing worth it? Is a testing platform worth it?
My answer is yes and no. On a theoretical level, yes, of course testing is worth it! Testing reveals insights about your site, it empowers you to ask questions to your customer and answer hypotheses that you would otherwise wonder about and internally debate! Testing also helps you increase performance on your site, thereby giving you the ability to spend more money on acquisition. However, testing also takes time. More importantly, it takes real, thoughtful, strategic intelligence. If you half-ass your testing and don’t have the time to come up with great ideas and also act on the results, you will not get your ROI.
Back at Offermatica, we used to liken a testing platform to a gym. You could belong to the best gym in the world, but if you don’t get your ass into it, you won’t lose weight. It’s plain and simple. Likewise, if you don’t exercise your site, you won’t see the lift. Garbage in, garbage out.
Testing also takes time and traffic. If you’re a company that’s just starting out, I question whether testing is the best place to spend your time and money. LunaMetrics has a nice post on just this topic. I think you should have a threshold of at least 50 conversion events per day before beginning to test. I look at testing as a great way to maximize your ad spend. If I’m spending a hundred dollars to acquire 100 people at $1 per head, increasing my conversion rate by 50% means I can now acquire 150 people for the same spend. That’s a worthwhile effort if I can test quickly and get to that answer in a couple weeks. I can then decide whether to spend more money acquiring traffic through my optimized funnel now, or re-allocate my spend to other channels.
But I’ve also talked to a lot of companies that are running lean and mean, just like we are at eduFire! If you’re one person in charge of all your marketing efforts, you probably don’t have time to run more than 1 or 2 tests a month. In that case, you really have to focus on prioritizing test ideas and going with strong hypotheses on high-traffic pages to maximize your effort. Would I recommend buying a testing platform? Probably not, unless you have the budget that you’re willing to dedicate towards hiring a resource to manage it. A platform does not run by itself, even the ones who say they do! At a minimum, you still need to load that platform up with stellar alternatives to run against whatever it is that you’re trying to improve. Rinse, lather and repeat.
A strategy of throwing mediocre stuff against the wall and seeing what sticks is not a strategy at all. It’s more like going to the gym once a month and walking on the treadmill for 20 minutes. I suppose there’s some value to getting your heart rate up, but you might not get the results you’re hoping for by the time bikini season rolls around.
What I’m trying to say is that testing *is* worth it, but only if you make the investment. And your investment cannot be just the platform, regardless of whether the platform is free or paid. Yes, more sophisticated platforms like Omniture Test&Target cost money, but I would wager that the greater “cost” is the time required to rally the organization around optimization and understand both the technical and strategic intricacies of whatever platform you are adopting. These are inherent whether you are using a free platform like Google Website Optimizer or not. (This is not to detract from GWO, I think it’s a great option for companies who are prepared to begin testing!)
Maybe one of the most important questions to ask yourself in determining whether it’s time to start testing is to investigate the amount you’re spending on acquisition. If you are spending a significant amount, then you absolutely should begin testing to maximize your budget most effectively. The other thing to look at is the amount of traffic your site is receiving. If you’re getting a lot of organic traffic, more power to you. Now optimize your funnel so you can begin spending money on other acquisition efforts as well!
So remember, testing is not a free ticket to lift. You have to work at it, just like everything else. But the benefits of working on the site spread to all of your efforts off the site as well. Now that’s a really good reason to begin testing!
so i have to say here that all this talk about testing is both interesting and confusing. it’s interesting cuz i run a qa team. it’s confusing because i think you’re only talk about a couple types of testing: performance and usability. ‘testing’ software and/or hardware involves a lot of different methodologies.
that testing thursdays thing is also confusing unless you’re only talking about testing small-scale consumer-facing websites for performance and usability. i got that right, right?
@mike – yes, ‘testing’ and ‘optimization’ are unbelievably broad terms, i’m talking about the type of testing that marketers do on the site to increase leads, sales, etc. The simplest example would be something like testing a red button instead of a blue button on a sign-up page.
And yes, the Testing Thursdays event is really geared towards marketers, but not necessarily small-scale sites. Big companies like Dell, Sears, and Allstate are running tests all the time!
Usability testing is a little different because that’s usually measured qualitatively, whereas conversion testing is measured quantitatively and relies on testing against a proper sample size population.
Hope that makes things (slightly) more clear