The most dismaying article I have read in recent months is The A/B Test: Inside the Technology that is Changing Business in Wired. So here is the basic idea. Brian Christian describes a process of changing updating websites that relies of data rather than what the designers or authors think is "right," "beautiful," or "useful." Here is the basic idea. Rather than make design and layout decisions based on hunches or design principles, the point is the make the decisions based on how users actually use the website. I know, I know, .. use data to actual make decisions... seems elementary, right.
My experience in developing websites in academic settings is that most people don't even know that you can collect analytic data on websites and most academic staff wouldn't know what to do with analytic data. The typical academic website rarely even goes through any basic user testing to see how it works. Yikes.
The A/B test is simple. There are numerous ways to design websites and multiple of ways of organizing information. The A/B Test idea is the try things out and see what works. Compare design A with design B and see what users like by randomly presenting some users with one version and other views with other versions. Over time this approach this approach will refine the design of the website and provide a means to develop a design that fits the user rather than the designers.
Here are the basic principles that Christian outlines:
1. Choose Everything instead of having to make choices.
2. Data makes the call rather than the person at the top makes the call.
3. The risk is making only tiny improvements rather than the risk is making a huge mistake.
4. Data can make the very idea of lessons obsolete rather than experience teaches us lessons.