[theme-reviewers] theme approval turnaround

Tue Oct 4 19:17:42 UTC 2011

I've tried to suggest the idea of implementing Selenium automated script
that actually browses an install by all the pages from the test data and
scrolls so it actually could be played automatically and it could be done
for like 2min only to see if there are any crucial visual unappealing
sectors. It would be nice to have, but someone has to create the script (the
scrolling in Selenium is a bit more tricky due to some JS things to be done
there).

I had exchanged a couple of mails with another theme author but was unable
to proceed further.

If there is anyone here with a Selenium exp, we could continue working in
this direction. When I get back home next week I'll probably ask in the
freelance networks for how much would that cost to be developed as a tool
(bad part is that I have to spend some time explaining the assignment
itself).

Mario Peshev
Training and Consulting Services @ DevriX
http://www.linkedin.com/in/mpeshev
http://peshev.net/blog

On Tue, Oct 4, 2011 at 10:11 PM, Kirk Wight <kwight at kwight.ca> wrote:

> I would imagine a method like this would be impacted by the inconsistency
> of available volunteers - the system could grind to a halt if the style.css
> people are unavailable for a period. If everyone knows what they are doing,
> the process will continue beginning to end regardless of the number of
> volunteers available at any given time.
>
> Kirk
>
> On 4 October 2011 15:06, Mikkel W. Breum <mikkel at wpkitchen.com> wrote:
>
>> Hi all,
>>
>> So far I've just reviewed one theme, so I'm still to learn a lot and get
>> the hang of it. I know that any new job takes some time to learn, and once
>> you get the routine everything goes much faster. However I can't help
>> feeling that the entire process could perhaps be optimized somewhat. I'm not
>> so much thinking about the parts that re/can be automated here, but the part
>> that has to be done by us flesh and blood testers.
>>
>> I mean, if it was a factory, the entire review process would be cut up
>> into individual parts, and different workers would be assigned a smaller
>> task to repeat. So for the WPTR process, that would look something like one
>> person being given 30 themes, and the person should then only check the
>> style.css for license, and URI compliance, and use a predefined tool to
>> check some boxes on what checks failed to pass. Another person would do the
>> discreet job of verifying a specific unit test, and so on.
>>
>> I can see that implementing this would require some technical changes to
>> how the review process is handled, but i'm sure it could boost review
>> performance considerably.
>>
>> What do you think about this? Would it be interesting to develop this idea
>> further?
>>
>> I have more ideas on how this could be implemented, also regarding some
>> tools that could be developed for the processing, but first I'd like to hear
>> what you all think about the thought?
>>
>> Mikkel
>>
>>
>>
>> On 04/10/2011, at 19.54, Chip Bennett wrote:
>>
>> The uploader script is a combination of the Theme Check Plugin checks, as
>> well as some (not-so) "secret sauce" tests (mostly security-related). The
>> uploader runs all of the Theme-Check checks, and prevents upload of any
>> Theme that has a WARNING or REQUIRED issue flagged.
>>
>> Actually, around the WordPress 3.1 release, we widened the moat
>> considerably with the uploader, which previously only *reported* the Theme
>> Check issues, but didn't act on them. We did see a momentary dip in
>> submitted Themes, but the numbers crept right back up.
>>
>> We've automated *just* about everything feasible, though I'm sure Pross
>> would love your help and input, if you have any ideas.
>>
>> Anything that we can get into Theme-Check can only help developers and
>> reviewers alike; however, some things simply require a human review. Theme
>> Check can check for, say, pairing of add_theme_support( 'post-thumbnails' )
>> and calls to the_post_thumbnail(), but it can't ensure that
>> the_post_thumbnail() is implemented correctly. Theme Check can ensure that
>> License/License URI header tags are included in style.css, but it can't
>> verify that the declared license is GPL-compatible, or that the supplied URI
>> is valid. Theme Check can ensure that appropriate template tags are
>> included, but it can't really duplicate the thoroughness of a human eye
>> reviewing the Theme as-installed, and going through the Theme Unit Tests.
>>
>> Assuming that the vast majority of Themes are submitted by developers who
>> fully desire and intend to adhere to the guidelines (I am an optimist, after
>> all), the best thing we can do for such developers is to provide a complete,
>> thorough review of their initial submission. But, we must balance that
>> against the expediency of the reviews.
>>
>> Chip
>>
>> On Tue, Oct 4, 2011 at 11:54 AM, Kirk Wight <kwight at kwight.ca> wrote:
>>
>>> Chip, I don't know you, but I already love you. WordPress loves you :)
>>>
>>> I did a quick check of recent tickets closed-not-approved, and it seems a
>>> lot of themes are rejected for not meeting basic requirements, and/or best
>>> practices. I'm sure it's been mentioned before, but cutting off more of
>>> these themes at the gate (the upload point) would avoid a lot of wasted time
>>> on reviewers' part, and reduce the number of tickets per approved theme.
>>>
>>> How does the upload check work, is it just a variation on the Theme Check
>>> plugin? Is there any way I/we can see what goes on there (SVN link maybe)?
>>> I'm not much of a PHP programmer, but if I can be of help to the person
>>> maintaining it by testing, etc, let me know.
>>>
>>> In the meantime, I'll review themes :)
>>>
>>> Kirk
>>>
>>>
>>>
>>> On 4 October 2011 11:58, Chip Bennett <chip at chipbennett.net> wrote:
>>>
>>>> The number of incoming tickets per month continues to climb steadily,
>>>> but slowly. We are up to approximately 12 tickets per day:
>>>> [image:
>>>> oimg?key=0AmhRfB-XJH5_dE5FU0tvcTF2MmZwaWVkSV9PWVVHbFE&oid=1&zx=96ihgzsxiv84]
>>>> And here is the monthly workload trend:
>>>> [image:
>>>> oimg?key=0AmhRfB-XJH5_dE5FU0tvcTF2MmZwaWVkSV9PWVVHbFE&oid=15&zx=am4vt98acw9k]
>>>> As you can see, the total number of reviewers per month is also trending
>>>> upward (note the spike, around the time that Justin Tadlock published his
>>>> blog post calling for help with Theme reviews); however, note the high
>>>> standard deviation, which in this case is indicative of a small number of
>>>> reviewers performing the bulk of the reviews in any given month.
>>>>
>>>> We can keep up with the incoming workload, if we average about 13
>>>> tickets closed per day.We're currently averaging 15 active Reviewers per
>>>> month, which would require an average of one ticket per day, per active
>>>> Reviewer.
>>>>
>>>> To improve that number, we either need to get each active Reviewer (the
>>>> bulk of whom review 5 or less tickets per month) to review considerably more
>>>> tickets per month, or else we need to get more Reviewers active per month.
>>>>
>>>> We have enough total Reviewers now that if everyone who has ever
>>>> reviewed a ticket would review one ticket per week, we would keep up with
>>>> the incoming workflow.
>>>>
>>>> My ultimate goal is to improve the percentage of approved tickets, and a
>>>> reduced number of tickets per Theme, which will drive down our total
>>>> workload. As you can see, this has been difficult:
>>>> [image:
>>>> oimg?key=0AmhRfB-XJH5_dE5FU0tvcTF2MmZwaWVkSV9PWVVHbFE&oid=16&zx=z105kgyojdh6]
>>>> We generally hold steady at about 20-25% of tickets  (not including
>>>> closed-newer-version tickets) being resolved as approved - though this past
>>>> month we did have a nice up-tick. I don't currently have a reliable means to
>>>> measure total number of tickets required per approval, although I would
>>>> really like to track this metric.
>>>>
>>>> So, that's where we are right now. As always, we're open to any ideas
>>>> for improvement!
>>>>
>>>> Chip
>>>>
>>>> On Tue, Oct 4, 2011 at 9:39 AM, Kirk Wight <kwight at kwight.ca> wrote:
>>>>
>>>>> That's a great idea - keeping that #2 queue as empty as possible would
>>>>> be a huge boost in consistency, from a users point of view.
>>>>>
>>>>> Out of curiosity, have you noticed any trends regarding increased
>>>>> submissions/workload? Maybe times of the month or year, or right after a
>>>>> WordPress point bump?
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> theme-reviewers mailing list
>>>>> theme-reviewers at lists.wordpress.org
>>>>> http://lists.wordpress.org/mailman/listinfo/theme-reviewers
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> theme-reviewers mailing list
>>>> theme-reviewers at lists.wordpress.org
>>>> http://lists.wordpress.org/mailman/listinfo/theme-reviewers
>>>>
>>>>
>>>
>>> _______________________________________________
>>> theme-reviewers mailing list
>>> theme-reviewers at lists.wordpress.org
>>> http://lists.wordpress.org/mailman/listinfo/theme-reviewers
>>>
>>>
>> _______________________________________________
>> theme-reviewers mailing list
>> theme-reviewers at lists.wordpress.org
>> http://lists.wordpress.org/mailman/listinfo/theme-reviewers
>>
>>
>>
>> _______________________________________________
>> theme-reviewers mailing list
>> theme-reviewers at lists.wordpress.org
>> http://lists.wordpress.org/mailman/listinfo/theme-reviewers
>>
>>
>
> _______________________________________________
> theme-reviewers mailing list
> theme-reviewers at lists.wordpress.org
> http://lists.wordpress.org/mailman/listinfo/theme-reviewers
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wordpress.org/pipermail/theme-reviewers/attachments/20111004/fcc23492/attachment-0001.htm>