Yeah, getting peer reviewed definitely isn’t a joke. I just think it’s important that people take the time to read and understand a study and its limitations, environment, etc, even peer reviewed articles. And that people report that information accurately and fairly to clients and on blogs. Overall, people are too quick in wanting to be “first” that they get a study’s gist and run with it. I’d just want to caution people to slow down and check the sources. I have no doubts that the people running the study and the study have faced lots of scrutiny, and I think the people who re-report the findings ought to face some scrutiny too.

(And I guess I was also talking about “studies” and “findings”–sorry I’d be using italics instead of quotes if I knew how–that are much less regulated and not peer reviewed. A lot of A/B test websites are popping up, and it’s not that they’re bad, they’re just very inconclusive which should be promoting discussion not asserting numbers as facts.)

As far as actual testing goes, looks like I wasn’t clear enough. When I go off on a tangent about being for an individual client, an individual site, individual audiences, etc, I meant that things should be tested for are all these individual variables. Not just to dismiss the client because a study doesn’t fit their website, but rather to emphasize the need to test for that client’s website to get the most accurate results. Of course, I also live in an ideal world with perfect clients and an excess of money and time.

I think that the Verification step is the one I see most often skipped. People simply accept numbers/figures because “so-and-so said it’s so!” As Chris says above, these things are often reviewed by others, but you need to know what that process was in each case before you simply accept data as-is.

Like you say, these things need to be brought down the practical, individual level. Testing, testing, and more testing.

Also, a good usability study is a little more complicated than your average middle school science fair project

I think your discussion is missing one important part of the equation: actual testing. You can argue back and forth the merits of a particular piece of research with a client until you turn blue in the face, but the ultimate argument is always going to be testing with real-life users. A smart client gives you time to do that.