[theme-reviewers] theme approval turnaround

Chip Bennett chip at chipbennett.net
Tue Oct 4 17:54:15 UTC 2011

The uploader script is a combination of the Theme Check Plugin checks, as
well as some (not-so) "secret sauce" tests (mostly security-related). The
uploader runs all of the Theme-Check checks, and prevents upload of any
Theme that has a WARNING or REQUIRED issue flagged.

Actually, around the WordPress 3.1 release, we widened the moat considerably
with the uploader, which previously only *reported* the Theme Check issues,
but didn't act on them. We did see a momentary dip in submitted Themes, but
the numbers crept right back up.

We've automated *just* about everything feasible, though I'm sure Pross
would love your help and input, if you have any ideas.

Anything that we can get into Theme-Check can only help developers and
reviewers alike; however, some things simply require a human review. Theme
Check can check for, say, pairing of add_theme_support( 'post-thumbnails' )
and calls to the_post_thumbnail(), but it can't ensure that
the_post_thumbnail() is implemented correctly. Theme Check can ensure that
License/License URI header tags are included in style.css, but it can't
verify that the declared license is GPL-compatible, or that the supplied URI
is valid. Theme Check can ensure that appropriate template tags are
included, but it can't really duplicate the thoroughness of a human eye
reviewing the Theme as-installed, and going through the Theme Unit Tests.

Assuming that the vast majority of Themes are submitted by developers who
fully desire and intend to adhere to the guidelines (I am an optimist, after
all), the best thing we can do for such developers is to provide a complete,
thorough review of their initial submission. But, we must balance that
against the expediency of the reviews.


On Tue, Oct 4, 2011 at 11:54 AM, Kirk Wight <kwight at kwight.ca> wrote:

> Chip, I don't know you, but I already love you. WordPress loves you :)
> I did a quick check of recent tickets closed-not-approved, and it seems a
> lot of themes are rejected for not meeting basic requirements, and/or best
> practices. I'm sure it's been mentioned before, but cutting off more of
> these themes at the gate (the upload point) would avoid a lot of wasted time
> on reviewers' part, and reduce the number of tickets per approved theme.
> How does the upload check work, is it just a variation on the Theme Check
> plugin? Is there any way I/we can see what goes on there (SVN link maybe)?
> I'm not much of a PHP programmer, but if I can be of help to the person
> maintaining it by testing, etc, let me know.
> In the meantime, I'll review themes :)
> Kirk
> On 4 October 2011 11:58, Chip Bennett <chip at chipbennett.net> wrote:
>> The number of incoming tickets per month continues to climb steadily, but
>> slowly. We are up to approximately 12 tickets per day:
>> [image:
>> oimg?key=0AmhRfB-XJH5_dE5FU0tvcTF2MmZwaWVkSV9PWVVHbFE&oid=1&zx=96ihgzsxiv84]
>> And here is the monthly workload trend:
>> [image:
>> oimg?key=0AmhRfB-XJH5_dE5FU0tvcTF2MmZwaWVkSV9PWVVHbFE&oid=15&zx=am4vt98acw9k]
>> As you can see, the total number of reviewers per month is also trending
>> upward (note the spike, around the time that Justin Tadlock published his
>> blog post calling for help with Theme reviews); however, note the high
>> standard deviation, which in this case is indicative of a small number of
>> reviewers performing the bulk of the reviews in any given month.
>> We can keep up with the incoming workload, if we average about 13 tickets
>> closed per day.We're currently averaging 15 active Reviewers per month,
>> which would require an average of one ticket per day, per active Reviewer.
>> To improve that number, we either need to get each active Reviewer (the
>> bulk of whom review 5 or less tickets per month) to review considerably more
>> tickets per month, or else we need to get more Reviewers active per month.
>> We have enough total Reviewers now that if everyone who has ever reviewed
>> a ticket would review one ticket per week, we would keep up with the
>> incoming workflow.
>> My ultimate goal is to improve the percentage of approved tickets, and a
>> reduced number of tickets per Theme, which will drive down our total
>> workload. As you can see, this has been difficult:
>> [image:
>> oimg?key=0AmhRfB-XJH5_dE5FU0tvcTF2MmZwaWVkSV9PWVVHbFE&oid=16&zx=z105kgyojdh6]
>> We generally hold steady at about 20-25% of tickets  (not including
>> closed-newer-version tickets) being resolved as approved - though this past
>> month we did have a nice up-tick. I don't currently have a reliable means to
>> measure total number of tickets required per approval, although I would
>> really like to track this metric.
>> So, that's where we are right now. As always, we're open to any ideas for
>> improvement!
>> Chip
>> On Tue, Oct 4, 2011 at 9:39 AM, Kirk Wight <kwight at kwight.ca> wrote:
>>> That's a great idea - keeping that #2 queue as empty as possible would be
>>> a huge boost in consistency, from a users point of view.
>>> Out of curiosity, have you noticed any trends regarding increased
>>> submissions/workload? Maybe times of the month or year, or right after a
>>> WordPress point bump?
>>> _______________________________________________
>>> theme-reviewers mailing list
>>> theme-reviewers at lists.wordpress.org
>>> http://lists.wordpress.org/mailman/listinfo/theme-reviewers
>> _______________________________________________
>> theme-reviewers mailing list
>> theme-reviewers at lists.wordpress.org
>> http://lists.wordpress.org/mailman/listinfo/theme-reviewers
> _______________________________________________
> theme-reviewers mailing list
> theme-reviewers at lists.wordpress.org
> http://lists.wordpress.org/mailman/listinfo/theme-reviewers
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wordpress.org/pipermail/theme-reviewers/attachments/20111004/285e40fa/attachment-0001.htm>

More information about the theme-reviewers mailing list