User Usability Testing

The goal of this exercise is to get input from a user about the applications being tested. The test works by putting the user in front of the application, giving them a task and letting them execute it, guided by the developer asking questions.

Process

The process works as follows: The process works as follows: The user is put in front of the opening screen of the application/tool. The developer or usability expert guiding the process gives them a goal, something like 'play a song from Cold Play' or 'connect to the wireless network'. Then the following protocol is executed by the person guiding the process, asking the user:

What is your first impression?
What do you think you can do here?
What would you do to reach your goal?
What do you expect to happen when you do it?
Please do it... (let user execute action)
Did it do what you expected?
If not: was it better or worse?
If task is not completed yet, go back to 1

If the task is finished, asked what the user thought of it all, discuss improvements etc.

Notes:

This kind of testing is very good to find grave issues in applications; it is not very good at fine tuning the interaction with your application. Be a bit wary of the feedback from a single user: repeat the process at least once with somebody else to make sure that what is an issue for one person is really a problem. Ideally, run the test a few times with various people to increase the reliability of the feedback.
Try to closely stick to the protocol, not skipping steps. Of course, if a user has already answered a question in his answer on another one, there is no need to ask that question again...
make clear that the user is ONLY to execute a SINGLE action in step 5. This means that most of the time, you are checking the expectations and ideas of the user instead of watching him/her clicking around.
As guide you're not supposed to give any hints as to what button the user should click or what he/she should do, other than perhaps remind him/her of the end goal he/she has been given
Despite the above point, there is of course no reason to let the user helplessly waste his/her time. If the user get stuck, help them to get unstuck.
Yes, it can be a bit hard to stay disciplined, not giving hints or telling the user what to do... And to stick to the protocol instead of just quickly letting the user click to the end. Try to keep to the structure anyway, it works best.
Extra tip: it is very valuable to record the sessions. Often you see things later on that you were not aware off during the session...

Results

We tested KScreen. I turned out to be very confusing ;-)
Also Plasma Media Center got tried out. See here for PMC feedback.
The network manager plasmoid session was even put on video: part 1 and part 2