However, it should always be noted that almost all testing is artificial and rarely replicates the impact on usability of real-life distractions. People “perform” when you test them and they are generally in some sort of closed environment that requires they puzzle through the task as opposed to what they do in real life at the first sign of trouble — which is bail and go somewhere else.

I find that usability testing is a great predictor of what designs will FAIL and a not-very-accurate predictor of what will succeed.

Here’s a couple of tips:
1. Don’t just accept ANY success as success. If it took the user more than 4-5 clicks, if they hesitated for more than a couple of seconds, used the back button or encountered an error before ultimately succeeding, count them as a failure — they won’t hang in through all that in real life.
2. Go back 3 months after you launch a new feature or flow and check your BI data to see how well it’s performing. Check again in 6 months. Look at what your testing predicted and see if it’s actually played out in the live environment. Be rigorous about this and you will quickly learn if your testing is providing an accurate view of success.
3. Do not have the people who designed the UX test it. No matter how much we all like to believe we can set aside pride of authorship and remain objective for the good of the overall project, it’s almost impossible to do.
4. Keep your test scenarios as general as possible and be careful as to how you tee-up the task. You can influence the outcome if you give away entry point info that a real-life user wouldn’t have and create false-positive results.

Great rad.

For what it’s worth, this is why prototyping in almost standard practice in our industry. Even though every project is different, as long as we’re iterative and fflexible, the testing should happen as a by-product of good design instincts.

Any way, keep up the great work, James; articles like this keep the discussion going in the right direction.

I’m glad you mentioned accessibility – albeit only in one place. This is an oft-overlooked area that can make such a difference.

Usability is always a funny topic – it will work better in different ways for different groups, on different projects. While I’m sad it hasent worked for most, it has certainly had a good outcome for me.