In any case, even if a usability test resolves a dispute, it doesn't do it in any kind of a statistically valid way. Unless you test thousands of people from all walks of life under all kinds of conditions, something that not even Microsoft can afford to do, you are not actually getting statistically meaningful results. Remember, the real strength of usability tests is in finding truffles—finding the broken bits so you can fix them. Actually looking at the results as if they were statistics is just not justified.

Author: Joel Spolsky