The Best of Both Worlds: Quantitative and Qualitative Testing for Web Design

In our previous post, we talked about Growth Driven Design and how it drives my decision when it comes to building, redesigning or improving websites.

I mentioned SumoMe as my weapon of choice in figuring out the best strategy to implement my projects.

So in this article, I’d like to give you an in-depth view on how I collect these data and present it in such a way that it is credible, fool-proof, and actionable.

Here’s my belief:

It doesn’t matter whether you have the most sophisticated data available. I don’t care if you give me the best metrics to get the job done. If they can’t give me the slightest hint about my customer’s behavior, it doesn’t make sense to gather them in the first place. Likewise, opinions don't matter because most of them are subjective anyway. We need facts based on measurable, unbiased results.

Analysis paralysis.

We all want to avoid this dreaded disease.

And as a web and marketing consultant, it’s my job to help my customers choose the right decisions based on information I’m showing them.

Which leads to our topic: Quantitative and Qualitative testing.

The former measures honest-to-goodness numbers, metrics, and analysis.

The latter gets results for your users – an experience you cannot get from numbers.

Now, let me give you a quick overview about these methods and how I use them.

Quantitative Testing

Quantitative testing is used for making small changes to a site with a larger volume of testers. It could be something like changing the text on a CTA, or changing the content on a page, or changing the whole layout of a page. These tests are automated; with data on each version being collected to make better design and content decisions later on.

Quantitative testing involves the following methods:

• A/B testing – Otherwise known as “split” testing, is a method of comparing two versions of a web page to find out which one performs better.
• Iteration – A design procedure where you subject your site to a cyclical process of prototyping, testing, and analyzing. This means, your design today may not be the same next month.
• Site traffic analysis – You measure how many people visit your website and how long they stay before heading to the exit. This metric is important to search engines as well – the more staying power your website has, the better ranking you’ll get.

Quantitative testing answers the following questions:

• Who are my visitors (E.g. mobile users, laptop/desktop users, MAC or PC users)?
• What or where did they came from (E.g. PPC ads, social media channels, organic searches)?
• When did they visit my site?

Quantitative testing allows me to present historical facts and trends. It provides me with concrete ideas on how to proceed with the design. I usually run it to measure a large data set – which occurs in big companies.

For example, I set up a heatmap tool to measure the clicks or impressions on my homepage. I set a date to gather data – about 1 to 2 weeks.

This is what a heatmap looks like:

I ran this test by setting my heatmap tool to detect up to 2,000 clicks.

I’m telling you, 2,000 clicks is too much data to monitor manually. I need a huge sample size to work with; thus, resorting to quantitative testing.

But it’s not enough …

Although quantitative testing has its perks, you still have to ask yourself “Why are my customers clicking this menu?”

Or, “How come most visitors click the red button instead of the orange one?”

Remember, the results you get from quantitative testing are just numbers. You will not get the true reason behind your findings.

You might guess or hypothesize and come up with a quick conclusion. But it isn’t enough simply because you’re not your own customer.

You’re too attached to your website to form an objective, unbiased view of what’s going on.

Simply put, quantitative testing only tells you that … “Yeah, this red button gets 50% more clicks than the orange one.” Or, “Well, your homepage gets 3,686 page views per day – 48% higher than last month’s figure.”

But honestly, do you know why?

You have to get down in the trenches, reach out to your customers, and find out their experiences when they visit your site.

And to gather these data, we have to conduct…

Qualitative Testing

This approach allows you to run a test with a smaller number of users. They’re your ideal client. Your goal is to direct the website design and content to appeal to them. You need to define the test, what are you looking to change, and what are the questions about the site you need answered. You may have a number of open-ended questions that you need to analyze in order to make decisions.

Qualitative testing is for answering specific questions you have. It is more detailed and involved than simply changing a few words. Also, it requires you to figure out the type of content and design your ideal client responds to. It’s all about getting the nitty-gritty details of your buyer’s journey and molding the site to satisfy them.

Qualitative testing answers the following questions:

• How do my customers interact with my site?
• Why did my customers behave that way?

Here’s how I gather information:

• User interviews – Surveys, meetings, or any other form of interaction with your clients.
• Focus groups – Gather a group people who have no connection with your company. You allow them to share their opinion, beliefs, or goals – whether negative or positive – towards your design.
• Usability testing – Collect inputs on how users use your system. Typically, you give users tasks to perform while your team listens, watches, and take notes.

In my experience, qualitative testing is ideal for making changes to content, overall layout, and bigger changes. This is when you ask pointed questions of the viewers to see if the site does what it's supposed to do.

Wrap up

C-Suite Network CEO, Thomas White wrote an article about listening to customers. Entrepreneur.com published the article and I’d like to share a snippet to drive my point:

“This story epitomies the dilemma businesspeople are facing in the world of big data. Sure, they have information on customers. This information is piling up in all sorts of places. For the most part, business owners are desperately making sure they don't miss any opportunity to record every touch point customers have with the company. But then what?”

Yes, “Now what?”

Balance is key when making big decisions. In today’s business landscape, you can’t survive with just a handful of data. You need well-rounded business tools that will enable you to compete against the giants of the industry.

When I work with my clients, I always make sure that I get the figures and establish the facts. There’s no room for half-baked decisions – especially when we’re talking about building a fully-functional business website. It’s a painstaking, expensive endeavor to let our faulty conclusions dictate our course of action.

There should be a marriage between cold, hard numbers and genuine customer feedback.

No second guessing.

No false assumptions.