[wp-hackers] User Feedback and Testing

[Owen Winkler]

Ryan Boren wrote:
> On Sun, 2006-01-01 at 15:17 -0500, Elliotte Harold wrote:
>> I'm curious. Has any actual user testing been done on WordPress? In 
>> other words, has anyone put non-developer users in front of a WordPress 
>> system, asked them to perform tasks, and watched them to see how they 
>> do, and what confuses them? I suspect the results would be informative. 
>> There are some other problems I've noticed in the UI, but so far those 
>> are all common to 1.5 and 2.0.
> 
> wordpress.com always tracks the latest development, and we have many
> thousands of users on wp.com who love to give feedback.  They worked
> with all the iterations of the RTE and the inline uploader, and their
> suggestions shaped development.  The RTE is a major draw to wp.com.

Testing is the fundamental reason why I think there is so much churning 
on these topics, and it's not just the kind of bug/patch testing that 
we're accustomed to hearing about on these lists.

I've heard more than once that this is the best-tested version of 
WordPress yet.  Maybe more people have downloaded and tried pre-release 
versions of WordPress than before, and maybe there has been a lot of 
feedback from WP.com (none of which seems to be made public like it is 
in Trac, now that I think about it).  Does that imply that user testing 
was done?  Have we really asked uninvolved users what they think of the 
software?  Whether they think something could be improved?  Or are we 
basing what we think works on how well they receive features that they 
have no influence over?

People say that WordPress was released too quickly.  I don't think this 
is phrased correctly.  I think that an inadequate testing duration was 
allotted for those last minute patches that were applied.  Nevermind if 
every single patch works flawlessly - What kind of policy is it to 
release as large a project as this without letting the code burn-in on 
testers machines for a few days?

We could keep pushing back the release date forever because the bugs 
would never end.  We would never release a new version.  Yeah, I get the 
idea.  Likewise, we could push the release back two weeks and allow a 
burn-in of frozen code to make sure those last-minute patches are 
working as expected.  Or, we could push the release back a month and 
have more complete documentation.  Clearly, a four-day release delay was 
valuable to the support folks.  In a delay's worst case, if none of the 
remaining bugs are addressed during that period, you have two things you 
didn't have before:  1) A more thorough idea of what is wrong with the 
release for use by support.  2) A firm, organized release date.

Real products have development cycles, where development is broken into 
phases that allow for discussion of new feature requirements, 
development of planned new features, testing of those features, and 
providing documentation and support for those features upon release. 
These cycles are not necessarily periodic (as in using a regular release 
date), but if are planned properly can be assigned a best-guess timeline 
that can be used for planning useful things like press releases, site 
design updates, and support ramp-up.

With a project this large affecting so many people, it seems about time 
that the development cycle of WordPress becomes a bit more openly 
scripted.  Otherwise we can continue as is: Guessing at new features, 
wondering if the new features are addressing the needs of users, 
speculating on release dates at the expense of complete documentation 
and support, and watching a growing unease among the project's most 
ardent supporters.

My belief is that nothing positive can happen if nobody says anything. 
I hope I have contributed positively by airing my thoughts on this 
matter, and I hope that my thoughts are considered and reasoned before 
hasty or pat responses are returned.

Owen