95% of all statistics are made up.

The saying above is a tongue and cheek reference to a very real phenomenon: people’s willingness to fudge the truth in order to prove their point. If I’m arguing for something and say, it’s proven that 32% of people don’t like xyz, it’s harder for my opponent to argue otherwise. Statistics are powerful because people think of them as facts, as undeniable, set in stone truths. But statistics can easily be miscalculated, misrepresented and manipulated to serve an agenda.

Why does this matter to you? Clients often request such-and-such because the stats show it increases conversion. But as the designer, as the person hired by the client to be an expert in this field, I know that the stats aren’t all that they seem. And once in this spot, it’s not easy to persuade the client that the numbers are wrong. I imagine most designers and developers have been in a similar situation. And most importantly, maybe the stats aren’t “wrong” or misleading or manipulated, but that doesn’t mean that the actions they point to are absolutely “right” for your client.

From beginning to end: I’ve created this handy (simplified) diagram that explains the process and end result of usability studying:

The Study – Someone somewhere does a usability study using standard scientific protocols. Nothing complicated, just your standard 8th grade science class stuff: a hypothesis, a control environment, a variable, etc.

The Stats – Based on the results of the study, the creators create some statistics. I have no doubt that the creators are legit people, so the stats are accurate and well reported.

Publishing – The creators have to eat, so they write up some press releases, write up the study for publication and throw it to the wolves in hope for a little financial return and recognition.

Verification – This is the step I feel is getting missed. People should be reading the original studies in full to determine validity. Are there any missing holes? Anything that people should be aware of? Don’t take a study and statistics at face value. (Yes, I am on the cynical side of life, if you’re wondering.) If at all possible, people should run a similar study to see if the results are the same. Findings should be repeatable. If they are voilà, the findings/stats are legitimate and you should pass them along with fervor. And if they’re not, a healthy discussion about why should ensue. Running the study again is a win-win.

Public Support – This is when blogs take that press release or journal abstract and then rewrite it. And then another blog takes that post and regurgitates it and so on and so on. Until the results and statistics of the study are so far removed from the study itself that the numbers lose their value.

Action – This is the step that makes all the previous steps matter. This is when the client wants that button red and HUGE because he read a three paragraph blog that a big red button increased conversion 3% in usability/conversion study #123. This is also the step where, an informed expert knows the ins, outs, loopholes and shortcomings of study #123 (and not just the three paragraph blog) that she can say, wait, this is how YOUR button differentiates from the study’s button, this is how YOUR users differentiate from the study’s subjects, this is how YOUR site differentiates from the study’s site. Hopefully, resolving any conflict between her and her client, and between the user and the design.

What’s this all mean for you? That depends are you a designer or a client? As a client, I’d make sure that my designer knows what’s up. Not only because that’s what you pay them for, but also because it will set you at ease. You won’t need to bring up every small change that you’ve heard will increase your conversions or traffic, because your designer, a good designer, should already know them. If you’re a designer, you’re being hired because you’re an expert in your field. And if you’re doing web design, knowing how a user interacts with the web and how certain elements affect conversion or traffic is part of being an expert. Don’t let other people tell you what works and why–read those primary sources!

I’m a strong believer in two things:

1. Single usability studies don’t prove anything; they just beg the need for repeat studying and individualized studying,
2. It’s always about the individual client, the individual website, the individual webpage, the individual user and their needs in relation to one another.

Design and development should be driven by the user’s situation not the usability statistics. And 99% of all online users in the entire universe agree with me.

Coming up in future months: I’ll be delving into some specific usability studies and talking about the parameters, controls, statistical outcomes etc. (don’t worry, it won’t be overly scientific) to provide a well rounded look at some common web elements such as page length, forms, credibility, eCommerce checkouts etc. Check back each month!

AND if you have a usability study or question that you want me to delve into, please leave it in the comments!