[theme-reviewers] theme approval turnaround

Tue Oct 4 19:18:39 UTC 2011

Hi Mikkel,

Funny you should mention that! Very early on in our existence, the WPTRT
discussed just such a process. The idea bantered around back then was to
have a group that would do "general" reviews, then a group that would do
"security" reviews, and then a group that would do "aesthetic"/Unit Test
reviews. The consensus was, however, that we don't really have the
infrastructure in place to make such a workflow efficient or effective.

The reality is that we are fairly limited in our Trac workflow. Ultimately,
the time it would take for one person to do one thorough, quality review is
equal to, if not less than, the time it would take for that same ticket to
make its way through a multi-step workflow. But even without our
infrastructure limitation, we just don't have enough bodies tackling the
problem consistently enough. (Again: not a criticism; just a reality.)

But what I've discovered, since that early discussion, is that there is
great benefit in traversing the learning curve (and it can be a bit of a
curve, to be sure). I think we would lose some of the tangential benefit of
the WPTRT "experiment" if we segregated ourselves in that manner. As it is,
rather than have a handful of "security gurus" tackling security-related
reviews, *everyone* does security reviews - and in the process, *everyone*
becomes more knowledgeable about Theme security. (I count myself among such
people, as I knew little to nothing about Theme security before getting
involved with Theme reviews.)

Also, we get more developers thinking about and discussing issues of Theme
security, Theme quality, code/functionality best practices, etc. -
discussions that wouldn't take place nearly as often if fewer people were
looking at all aspects of Theme review.

So I think, at least for the time being, we gain more than we lose with the
current process.

Chip

On Tue, Oct 4, 2011 at 2:06 PM, Mikkel W. Breum <mikkel at wpkitchen.com>wrote:

> Hi all,
>
> So far I've just reviewed one theme, so I'm still to learn a lot and get
> the hang of it. I know that any new job takes some time to learn, and once
> you get the routine everything goes much faster. However I can't help
> feeling that the entire process could perhaps be optimized somewhat. I'm not
> so much thinking about the parts that re/can be automated here, but the part
> that has to be done by us flesh and blood testers.
>
> I mean, if it was a factory, the entire review process would be cut up into
> individual parts, and different workers would be assigned a smaller task to
> repeat. So for the WPTR process, that would look something like one person
> being given 30 themes, and the person should then only check the style.css
> for license, and URI compliance, and use a predefined tool to check some
> boxes on what checks failed to pass. Another person would do the discreet
> job of verifying a specific unit test, and so on.
>
> I can see that implementing this would require some technical changes to
> how the review process is handled, but i'm sure it could boost review
> performance considerably.
>
> What do you think about this? Would it be interesting to develop this idea
> further?
>
> I have more ideas on how this could be implemented, also regarding some
> tools that could be developed for the processing, but first I'd like to hear
> what you all think about the thought?
>
> Mikkel
>
>
>
> On 04/10/2011, at 19.54, Chip Bennett wrote:
>
> The uploader script is a combination of the Theme Check Plugin checks, as
> well as some (not-so) "secret sauce" tests (mostly security-related). The
> uploader runs all of the Theme-Check checks, and prevents upload of any
> Theme that has a WARNING or REQUIRED issue flagged.
>
> Actually, around the WordPress 3.1 release, we widened the moat
> considerably with the uploader, which previously only *reported* the Theme
> Check issues, but didn't act on them. We did see a momentary dip in
> submitted Themes, but the numbers crept right back up.
>
> We've automated *just* about everything feasible, though I'm sure Pross
> would love your help and input, if you have any ideas.
>
> Anything that we can get into Theme-Check can only help developers and
> reviewers alike; however, some things simply require a human review. Theme
> Check can check for, say, pairing of add_theme_support( 'post-thumbnails' )
> and calls to the_post_thumbnail(), but it can't ensure that
> the_post_thumbnail() is implemented correctly. Theme Check can ensure that
> License/License URI header tags are included in style.css, but it can't
> verify that the declared license is GPL-compatible, or that the supplied URI
> is valid. Theme Check can ensure that appropriate template tags are
> included, but it can't really duplicate the thoroughness of a human eye
> reviewing the Theme as-installed, and going through the Theme Unit Tests.
>
> Assuming that the vast majority of Themes are submitted by developers who
> fully desire and intend to adhere to the guidelines (I am an optimist, after
> all), the best thing we can do for such developers is to provide a complete,
> thorough review of their initial submission. But, we must balance that
> against the expediency of the reviews.
>
> Chip
>
> On Tue, Oct 4, 2011 at 11:54 AM, Kirk Wight <kwight at kwight.ca> wrote:
>
>> Chip, I don't know you, but I already love you. WordPress loves you :)
>>
>> I did a quick check of recent tickets closed-not-approved, and it seems a
>> lot of themes are rejected for not meeting basic requirements, and/or best
>> practices. I'm sure it's been mentioned before, but cutting off more of
>> these themes at the gate (the upload point) would avoid a lot of wasted time
>> on reviewers' part, and reduce the number of tickets per approved theme.
>>
>> How does the upload check work, is it just a variation on the Theme Check
>> plugin? Is there any way I/we can see what goes on there (SVN link maybe)?
>> I'm not much of a PHP programmer, but if I can be of help to the person
>> maintaining it by testing, etc, let me know.
>>
>> In the meantime, I'll review themes :)
>>
>> Kirk
>>
>>
>>
>> On 4 October 2011 11:58, Chip Bennett <chip at chipbennett.net> wrote:
>>
>>> The number of incoming tickets per month continues to climb steadily, but
>>> slowly. We are up to approximately 12 tickets per day:
>>> [image:
>>> oimg?key=0AmhRfB-XJH5_dE5FU0tvcTF2MmZwaWVkSV9PWVVHbFE&oid=1&zx=96ihgzsxiv84]
>>> And here is the monthly workload trend:
>>> [image:
>>> oimg?key=0AmhRfB-XJH5_dE5FU0tvcTF2MmZwaWVkSV9PWVVHbFE&oid=15&zx=am4vt98acw9k]
>>> As you can see, the total number of reviewers per month is also trending
>>> upward (note the spike, around the time that Justin Tadlock published his
>>> blog post calling for help with Theme reviews); however, note the high
>>> standard deviation, which in this case is indicative of a small number of
>>> reviewers performing the bulk of the reviews in any given month.
>>>
>>> We can keep up with the incoming workload, if we average about 13 tickets
>>> closed per day.We're currently averaging 15 active Reviewers per month,
>>> which would require an average of one ticket per day, per active Reviewer.
>>>
>>> To improve that number, we either need to get each active Reviewer (the
>>> bulk of whom review 5 or less tickets per month) to review considerably more
>>> tickets per month, or else we need to get more Reviewers active per month.
>>>
>>> We have enough total Reviewers now that if everyone who has ever reviewed
>>> a ticket would review one ticket per week, we would keep up with the
>>> incoming workflow.
>>>
>>> My ultimate goal is to improve the percentage of approved tickets, and a
>>> reduced number of tickets per Theme, which will drive down our total
>>> workload. As you can see, this has been difficult:
>>> [image:
>>> oimg?key=0AmhRfB-XJH5_dE5FU0tvcTF2MmZwaWVkSV9PWVVHbFE&oid=16&zx=z105kgyojdh6]
>>> We generally hold steady at about 20-25% of tickets  (not including
>>> closed-newer-version tickets) being resolved as approved - though this past
>>> month we did have a nice up-tick. I don't currently have a reliable means to
>>> measure total number of tickets required per approval, although I would
>>> really like to track this metric.
>>>
>>> So, that's where we are right now. As always, we're open to any ideas for
>>> improvement!
>>>
>>> Chip
>>>
>>> On Tue, Oct 4, 2011 at 9:39 AM, Kirk Wight <kwight at kwight.ca> wrote:
>>>
>>>> That's a great idea - keeping that #2 queue as empty as possible would
>>>> be a huge boost in consistency, from a users point of view.
>>>>
>>>> Out of curiosity, have you noticed any trends regarding increased
>>>> submissions/workload? Maybe times of the month or year, or right after a
>>>> WordPress point bump?
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> theme-reviewers mailing list
>>>> theme-reviewers at lists.wordpress.org
>>>> http://lists.wordpress.org/mailman/listinfo/theme-reviewers
>>>>
>>>>
>>>
>>> _______________________________________________
>>> theme-reviewers mailing list
>>> theme-reviewers at lists.wordpress.org
>>> http://lists.wordpress.org/mailman/listinfo/theme-reviewers
>>>
>>>
>>
>> _______________________________________________
>> theme-reviewers mailing list
>> theme-reviewers at lists.wordpress.org
>> http://lists.wordpress.org/mailman/listinfo/theme-reviewers
>>
>>
> _______________________________________________
> theme-reviewers mailing list
> theme-reviewers at lists.wordpress.org
> http://lists.wordpress.org/mailman/listinfo/theme-reviewers
>
>
>
> _______________________________________________
> theme-reviewers mailing list
> theme-reviewers at lists.wordpress.org
> http://lists.wordpress.org/mailman/listinfo/theme-reviewers
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wordpress.org/pipermail/theme-reviewers/attachments/20111004/9c54d65a/attachment.htm>