Improve your usability testing note taking

Daniel Foster / Filterbrew (https://www.filterbrew.com)

Recording what happens during a usability test is a crucial task as your notes are the basis of your later analysis. We’ve been collecting a list of tips on taking notes based on our experience of years testing with users.

Ideally, two kinds of professionals should be involved in a user test: a moderator and a note taker.

In practice, due to budget constraints, many users tests are run with one facilitator who moderates the sessions, takes notes and writes the final report.

This makes note taking a quite difficult task and you may miss some important finding since you can’t focus your attention on the user and on your notes at the same time.

Whether you are in charge of everything or you are just taking notes, a well-structured template will indeed be one of your most important tools during a user test.

USE A TEMPLATE

Preparing a template for your notes is a task that some find time-consuming and boring.

However, a template requires only a little effort, it can save you plenty of time when writing the final report and it helps you to explain your findings in a more effective way. By adopting a template, we’ve managed to cut our reporting times by half.

A structured template has several advantages VS free-form writing:

• It guides you through the study, so you won’t forget to take notes on any important topic.
• You can easily retrieve a note on a specific aspect at any time after the study is over.
• It helps compare between participants.
• It makes it easier to calculate quant. data.

CHOOSE THE RIGHT FORMAT FOR YOUR NOTES

We usually take notes in our laptops. It’s faster, we don’t have to decipher our ugly handwriting and it helps maintain notes tidy.

We use a Google Drive spreadsheet which could be a good solution for most people. It allows to keep your notes well arranged and the autosave feature is definitely a plus that can save hours of work.

There is not a single template or format that can work for everybody. Our advice is to try different solutions and see which one makes you feel more comfortable. Be open to change or tweak it. Each usability study is different and a good note-taker template is often the result of incremental changes.

BUILD YOUR TEMPLATE AROUND THE TESTING SCRIPT

Let’s take a step back…

When you decide to run a user test you do it because you have some critical questions regarding your product or service. For example: are my customers able to use feature x? How many usability issues does my website have?

Whatever your critical questions are, your testing guide -which is what your participants will do during the study and the questions you will ask them- will be aimed to get answers to them.

For example, if you want to test whether your forms are easy to use, you will ask your participants to perform a task that requires filling in some forms. Or, if you want to discover the users’ opinion regarding a certain feature, you will ask some questions about that feature, how easy was for them to use it, how it could you improve it, etc…

Pretty obvious, indeed!

And It is pretty obvious too that your notes should record all the information needed to provide the answers you are looking for.

However, we’ve seen many usability testing reports that were not delivering any useful insight partly because the notetaker failed to record all the relevant info.

When you are preparing your template, make sure you are covering all the key aspects you are investigating.

Build your template around the testing script and make sure it allows you to record any relevant data that might not be included in the guide but is related to the study goals.

PERFORM A DETAILED INSPECTION OF WHAT YOUR ARE TESTING

Whether you were the one who prepared the testing script or you just received it from someone else, you should inspect the website/app you are testing prior to the test.

If you do it, you will probably find many aspects you believe will turn into an issue for some of your participants. Since you are a UX expert and, hopefully, you have ran many other user tests in the past, your predictions can be right and that means you will have to report those issues in your notes.

You should be ready to take notes about those aspects during the test by making room for them in your template.

DON’T RECORD MORE INFORMATION THAN YOU NEED

Years ago, we ran a user test with 36 participants over three different locations and, because of this almost quantitative approach, we prepared a long excel template in which we collected usage data of many interface elements participants could use during the test.

Did we use all of them for the report? No.

When we analyzed the results, we finally found that only 1/3 of them provided valuable insights on the overall user experience and could be used to enrich the explanation of the issues we had discovered.

Another 20% (Such as “How many times did they use the filters?”) was added to the report to provide a few quantitative data that were welcomed by the client but had little to contribute to the analysis.

The remaining 50% was simply a waste of time: we could have reduced by half the effort when taking notes during the test.

So, what did we do wrong?

• Forgot we were running a qualitative user test and, thus, should provide few, relevant, quantitative data such as success rates and the frequency of each problem.
• We thought that it might be useful for our client to have lots of usage data, while this is not the scope of a user test. Our client could have gathered more precise information with his analytics tool.
• We collected lots of data just because we were able to do it, not because we really needed it.

When you are preparing your template, make sure you are not recording more information that the one you really need. Once again, stick to the study goals and the technique you are using.

REDUCE THE AMOUNT OF DATA TO BE RECORDED

In the begining, our templates were structured by generic topics (such as “Forms”, “Search box”, “Error messages”) and, for each of them, we used to write a description of the problems encountered by each participant.

That was very inefficient for three reasons:

• You have to write a lot each time you discover an issue on a topic, explaining what happened and why. This meant that you are not able to pay too much attention to the user and sometimes have to check the videos from the sessions since you are not fast enough to write down everything.
• Many issues are recurrent, but you can describe the same issue in different ways. When you are analyzing the results, you have to read all your notes over and over again to spot the problems.
• Since your descriptions do not follow any standard, it is hard to quantify how many times the same error occurred.

We soon realized that, since some issues could be predicted in advance or were likely to happen, we could rather record them as a yes/no answer to a series of questions.

For example, the topic “error messages” can become three questions:

• Does the user see the error message?
• Does the user understand the error message?
• Is the user able to recover from the error?

Plus, of course, a “Comments/More” field available to provide further details.
By doing so, you are able to summarize what happened with error messages by using only a few words and you can see how often they occurred at a glance.

However, even if you do an extensive revision of the website you are testing, you should always foresee a space for any other issue that may arise during the sessions or for your comments.

TEST YOUR TEMPLATE

A note-taker template is a living document that can (and has to) be adapted to your discoveries while you are testing users. You may probably find out a recurring issue that you were not considering or, on the contrary, discover that some of the problems you were expecting did not happen at all.

Updating a template can be difficult while you are running a test, with little time available between a user and a client that wants to debrief after each session.

That’s why we suggest to have your template ready for the pilot session and not to built it as a result of it.

You will have the opportunity to put it to the test and make most of the changes immediately.

¡Good luck!

At TeaCup Lab we are experts in User Research. We test with real users to help you create usable products and great user experiences.

Tell us about your project and discover how we can help you.

© Photo by Daniel Foster / Filterbrew (https://www.filterbrew.com/)