[wp-hackers] User Feedback and Testing

[Elliotte Harold]

David House wrote:

> I think this will be a great alternative to automated testing.
> Although on the face of it it seems like we're going to need hundreds
> of people to fill this out, we really only need five or six per
> feature group, and the staffing can overlap so that we'd only need,
> say, 50 testers. This will be a comfortable expansion up from our
> current numbers and should really direct the bugfixing efforts.

This keeps getting mixed up with searching for and finding bugs. This 
isn't about that. Allow me to summarize. There are two separate issues 
here, both of which need to be addressed:

1. Bug hunting: finding areas where the code does not behave as it is 
intended too. This can be addressed by unit testing, static code 
analysis, edge condition analysis, code inspection, and various other 
techniques. This is often the domain of professional testers and does 
not require end user participation, though if an end user finds such a 
big, great. This sort of testing can and should be automated.

2. Usability testing: finding areas where the code does exactly what the 
programmer intended it to do, and yet this is the wrong thing to do. 
This is where we need end users, because we cannot anticipate all the 
areas where they are likely to encounter problems. The question here is 
not whether the user interface behaves as the programmer intended it to. 
It is whether the design of the user interface fells natural to the end 
user, and allows them to accomplish the overall tasks in a 
straightforward, obvious, and convenient manner. This sort of testing 
really can't be automated, though it can be formalized.

Another difference: bugs are often binary. Either the bug is present or 
it isn't. Usability is more of a continuum. It's a question of more or 
less, not yes or no.


-- 
Elliotte Rusty Harold  elharo at metalab.unc.edu
XML in a Nutshell 3rd Edition Just Published!
http://www.cafeconleche.org/books/xian3/
http://www.amazon.com/exec/obidos/ISBN=0596007647/cafeaulaitA/ref=nosim