[theme-reviewers] theme approval turnaround

Tue Oct 4 19:11:36 UTC 2011

I would imagine a method like this would be impacted by the inconsistency of
available volunteers - the system could grind to a halt if the style.css
people are unavailable for a period. If everyone knows what they are doing,
the process will continue beginning to end regardless of the number of
volunteers available at any given time.

Kirk

On 4 October 2011 15:06, Mikkel W. Breum <mikkel at wpkitchen.com> wrote:

> Hi all,
>
> So far I've just reviewed one theme, so I'm still to learn a lot and get
> the hang of it. I know that any new job takes some time to learn, and once
> you get the routine everything goes much faster. However I can't help
> feeling that the entire process could perhaps be optimized somewhat. I'm not
> so much thinking about the parts that re/can be automated here, but the part
> that has to be done by us flesh and blood testers.
>
> I mean, if it was a factory, the entire review process would be cut up into
> individual parts, and different workers would be assigned a smaller task to
> repeat. So for the WPTR process, that would look something like one person
> being given 30 themes, and the person should then only check the style.css
> for license, and URI compliance, and use a predefined tool to check some
> boxes on what checks failed to pass. Another person would do the discreet
> job of verifying a specific unit test, and so on.
>
> I can see that implementing this would require some technical changes to
> how the review process is handled, but i'm sure it could boost review
> performance considerably.
>
> What do you think about this? Would it be interesting to develop this idea
> further?
>
> I have more ideas on how this could be implemented, also regarding some
> tools that could be developed for the processing, but first I'd like to hear
> what you all think about the thought?
>
> Mikkel
>
>
>
> On 04/10/2011, at 19.54, Chip Bennett wrote:
>
> The uploader script is a combination of the Theme Check Plugin checks, as
> well as some (not-so) "secret sauce" tests (mostly security-related). The
> uploader runs all of the Theme-Check checks, and prevents upload of any
> Theme that has a WARNING or REQUIRED issue flagged.
>
> Actually, around the WordPress 3.1 release, we widened the moat
> considerably with the uploader, which previously only *reported* the Theme
> Check issues, but didn't act on them. We did see a momentary dip in
> submitted Themes, but the numbers crept right back up.
>
> We've automated *just* about everything feasible, though I'm sure Pross
> would love your help and input, if you have any ideas.
>
> Anything that we can get into Theme-Check can only help developers and
> reviewers alike; however, some things simply require a human review. Theme
> Check can check for, say, pairing of add_theme_support( 'post-thumbnails' )
> and calls to the_post_thumbnail(), but it can't ensure that
> the_post_thumbnail() is implemented correctly. Theme Check can ensure that
> License/License URI header tags are included in style.css, but it can't
> verify that the declared license is GPL-compatible, or that the supplied URI
> is valid. Theme Check can ensure that appropriate template tags are
> included, but it can't really duplicate the thoroughness of a human eye
> reviewing the Theme as-installed, and going through the Theme Unit Tests.
>
> Assuming that the vast majority of Themes are submitted by developers who
> fully desire and intend to adhere to the guidelines (I am an optimist, after
> all), the best thing we can do for such developers is to provide a complete,
> thorough review of their initial submission. But, we must balance that
> against the expediency of the reviews.
>
> Chip
>
> On Tue, Oct 4, 2011 at 11:54 AM, Kirk Wight <kwight at kwight.ca> wrote:
>
>> Chip, I don't know you, but I already love you. WordPress loves you :)
>>
>> I did a quick check of recent tickets closed-not-approved, and it seems a
>> lot of themes are rejected for not meeting basic requirements, and/or best
>> practices. I'm sure it's been mentioned before, but cutting off more of
>> these themes at the gate (the upload point) would avoid a lot of wasted time
>> on reviewers' part, and reduce the number of tickets per approved theme.
>>
>> How does the upload check work, is it just a variation on the Theme Check
>> plugin? Is there any way I/we can see what goes on there (SVN link maybe)?
>> I'm not much of a PHP programmer, but if I can be of help to the person
>> maintaining it by testing, etc, let me know.
>>
>> In the meantime, I'll review themes :)
>>
>> Kirk
>>
>>
>>
>> On 4 October 2011 11:58, Chip Bennett <chip at chipbennett.net> wrote:
>>
>>> The number of incoming tickets per month continues to climb steadily, but
>>> slowly. We are up to approximately 12 tickets per day:
>>> [image:
>>> oimg?key=0AmhRfB-XJH5_dE5FU0tvcTF2MmZwaWVkSV9PWVVHbFE&oid=1&zx=96ihgzsxiv84]
>>> And here is the monthly workload trend:
>>> [image:
>>> oimg?key=0AmhRfB-XJH5_dE5FU0tvcTF2MmZwaWVkSV9PWVVHbFE&oid=15&zx=am4vt98acw9k]
>>> As you can see, the total number of reviewers per month is also trending
>>> upward (note the spike, around the time that Justin Tadlock published his
>>> blog post calling for help with Theme reviews); however, note the high
>>> standard deviation, which in this case is indicative of a small number of
>>> reviewers performing the bulk of the reviews in any given month.
>>>
>>> We can keep up with the incoming workload, if we average about 13 tickets
>>> closed per day.We're currently averaging 15 active Reviewers per month,
>>> which would require an average of one ticket per day, per active Reviewer.
>>>
>>> To improve that number, we either need to get each active Reviewer (the
>>> bulk of whom review 5 or less tickets per month) to review considerably more
>>> tickets per month, or else we need to get more Reviewers active per month.
>>>
>>> We have enough total Reviewers now that if everyone who has ever reviewed
>>> a ticket would review one ticket per week, we would keep up with the
>>> incoming workflow.
>>>
>>> My ultimate goal is to improve the percentage of approved tickets, and a
>>> reduced number of tickets per Theme, which will drive down our total
>>> workload. As you can see, this has been difficult:
>>> [image:
>>> oimg?key=0AmhRfB-XJH5_dE5FU0tvcTF2MmZwaWVkSV9PWVVHbFE&oid=16&zx=z105kgyojdh6]
>>> We generally hold steady at about 20-25% of tickets  (not including
>>> closed-newer-version tickets) being resolved as approved - though this past
>>> month we did have a nice up-tick. I don't currently have a reliable means to
>>> measure total number of tickets required per approval, although I would
>>> really like to track this metric.
>>>
>>> So, that's where we are right now. As always, we're open to any ideas for
>>> improvement!
>>>
>>> Chip
>>>
>>> On Tue, Oct 4, 2011 at 9:39 AM, Kirk Wight <kwight at kwight.ca> wrote:
>>>
>>>> That's a great idea - keeping that #2 queue as empty as possible would
>>>> be a huge boost in consistency, from a users point of view.
>>>>
>>>> Out of curiosity, have you noticed any trends regarding increased
>>>> submissions/workload? Maybe times of the month or year, or right after a
>>>> WordPress point bump?
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> theme-reviewers mailing list
>>>> theme-reviewers at lists.wordpress.org
>>>> http://lists.wordpress.org/mailman/listinfo/theme-reviewers
>>>>
>>>>
>>>
>>> _______________________________________________
>>> theme-reviewers mailing list
>>> theme-reviewers at lists.wordpress.org
>>> http://lists.wordpress.org/mailman/listinfo/theme-reviewers
>>>
>>>
>>
>> _______________________________________________
>> theme-reviewers mailing list
>> theme-reviewers at lists.wordpress.org
>> http://lists.wordpress.org/mailman/listinfo/theme-reviewers
>>
>>
> _______________________________________________
> theme-reviewers mailing list
> theme-reviewers at lists.wordpress.org
> http://lists.wordpress.org/mailman/listinfo/theme-reviewers
>
>
>
> _______________________________________________
> theme-reviewers mailing list
> theme-reviewers at lists.wordpress.org
> http://lists.wordpress.org/mailman/listinfo/theme-reviewers
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wordpress.org/pipermail/theme-reviewers/attachments/20111004/1429d906/attachment.htm>