Online surveys and usability testing

When you are working professionally with online communications, you typically work with a conceptual separation between:

1. Usability of the communication
2. Relevance of the communication
3. Visual design of the communication

The separation makes sense for the professional because it makes it possible to

1. Optimize usability through improved Information Architecture
2. Optimize relevance through improved content strategy and planning
3. Optimize the design through an improved use of graphical elements

The problem about this separation is that it is only conceptual. It does not exist in real life and it certainly does not exist in the heads of ordinary users visiting websites. For the casual visitor of a website, there is no distinction between usability, relevance and design. If the website is really relevant to the user, she will typically also think that it is really easy to use and maybe even also good looking. Similarly a user visiting two websites with exactly the same content, but with different degrees of usability will also consider the user friendly website much more relevant than the website with low usability.
When we, as professionals, try to determine usability, relevance or design of a solution we therefore need to be very cautious about how we talk to the actual users of a website. If we ask about relevance, we may actually get an answer which is much more related to usability and vice versa.
This all may seem a bit abstract, so let’s provide an example.
In the early days of the Internet I was testing the e-banking solution of Danske Bank. We used qualitative as well as quantitative methods – an online survey and a think-aloud test. Our gut-feelings told us that the usability of this very early version of the online bank would be very low, and this gut-feeling was confirmed in the think-aloud test, where all of the first-time users more or less failed to understand the basics of the application in the 1,5 hours the test lasted. However, and this was the surprising part, in the online surveys we asked a number of usability questions about navigation and general user friendliness of the online bank, and this time the results were in the completely opposite end of the scale – people loved the online bank and found it very easy to use.
The reason, of course, was that the relevance of an online bank was tremendously high. So high that people were willing to spend hours and hours to actually learn how to use the application, and when they had learned how to navigate the not-so-user-friendly application, they actually found it quite easy to use. The users were simply unable to tell the difference between relevance of the online bank and the usability of the online bank.
This is approximately eight years ago, and I told myself that I would never again make the mistake to ask people usability questions in an online survey. If I want to test the usability of a website, I will either use a qualitative think-aloud test or actual tracking data using some kind of metrics system.
Unfortunately it seems like my ancient experience has not become common lore in the web analytics business. More and more often I seem to bump into online surveys on all sorts of websites that are trying to ask people questions about the usability of the website (see example below).
The problem about this kind of surveys is that they simply do not provide the answers that the website owners think they are getting. It may seem like an easy and cheap solution when you buy the survey from the web analytics consultants, and it may seem like you are actually getting answers to your questions, but this is unfortunately not the case!
Online surveys are utterly useless when it comes to determining usability of a website. They are extremely good at determining demographics and content relevance questions, but they are simply unable to give you truthful answers when it comes to the usability of your website.
This may sound very dogmatic and not very pragmatic in the face of real life, where you are often constrained by time and resources. However, being a bit dogmatic when it comes to research methodology may in the long run be more pragmatic, because you will avoid getting the wrong answers to the right questions.