Hi Mikkel,<div><br></div><div>Funny you should mention that! Very early on in our existence, the WPTRT discussed just such a process. The idea bantered around back then was to have a group that would do "general" reviews, then a group that would do "security" reviews, and then a group that would do "aesthetic"/Unit Test reviews. The consensus was, however, that we don't really have the infrastructure in place to make such a workflow efficient or effective.</div>
<div><br></div><div>The reality is that we are fairly limited in our Trac workflow. Ultimately, the time it would take for one person to do one thorough, quality review is equal to, if not less than, the time it would take for that same ticket to make its way through a multi-step workflow. But even without our infrastructure limitation, we just don't have enough bodies tackling the problem consistently enough. (Again: not a criticism; just a reality.) </div>
<div><br></div><div>But what I've discovered, since that early discussion, is that there is great benefit in traversing the learning curve (and it can be a bit of a curve, to be sure). I think we would lose some of the tangential benefit of the WPTRT "experiment" if we segregated ourselves in that manner. As it is, rather than have a handful of "security gurus" tackling security-related reviews, *everyone* does security reviews - and in the process, *everyone* becomes more knowledgeable about Theme security. (I count myself among such people, as I knew little to nothing about Theme security before getting involved with Theme reviews.)</div>
<div><br></div><div>Also, we get more developers thinking about and discussing issues of Theme security, Theme quality, code/functionality best practices, etc. - discussions that wouldn't take place nearly as often if fewer people were looking at all aspects of Theme review.</div>
<div><br></div><div>So I think, at least for the time being, we gain more than we lose with the current process.</div><div><br></div><div>Chip</div><div><br><div class="gmail_quote">On Tue, Oct 4, 2011 at 2:06 PM, Mikkel W. Breum <span dir="ltr"><<a href="mailto:mikkel@wpkitchen.com">mikkel@wpkitchen.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div style="word-wrap:break-word">Hi all, <div><br></div><div>So far I've just reviewed one theme, so I'm still to learn a lot and get the hang of it. I know that any new job takes some time to learn, and once you get the routine everything goes much faster. However I can't help feeling that the entire process could perhaps be optimized somewhat. I'm not so much thinking about the parts that re/can be automated here, but the part that has to be done by us flesh and blood testers.</div>
<div><br></div><div>I mean, if it was a factory, the entire review process would be cut up into individual parts, and different workers would be assigned a smaller task to repeat. So for the WPTR process, that would look something like one person being given 30 themes, and the person should then only check the style.css for license, and URI compliance, and use a predefined tool to check some boxes on what checks failed to pass. Another person would do the discreet job of verifying a specific unit test, and so on.</div>
<div><br></div><div>I can see that implementing this would require some technical changes to how the review process is handled, but i'm sure it could boost review performance considerably. </div><div><br></div><div>What do you think about this? Would it be interesting to develop this idea further?</div>
<div><br></div><div>I have more ideas on how this could be implemented, also regarding some tools that could be developed for the processing, but first I'd like to hear what you all think about the thought?</div><div>
<br></div><font color="#888888"><div>Mikkel</div></font><div><div></div><div class="h5"><div><br></div><div><br></div><div><br><div><div>On 04/10/2011, at 19.54, Chip Bennett wrote:</div><br><blockquote type="cite"><div>The uploader script is a combination of the Theme Check Plugin checks, as well as some (not-so) "secret sauce" tests (mostly security-related). The uploader runs all of the Theme-Check checks, and prevents upload of any Theme that has a WARNING or REQUIRED issue flagged.</div>
<div><br></div>Actually, around the WordPress 3.1 release, we widened the moat considerably with the uploader, which previously only *reported* the Theme Check issues, but didn't act on them. We did see a momentary dip in submitted Themes, but the numbers crept right back up.<div>
<br></div><div>We've automated *just* about everything feasible, though I'm sure Pross would love your help and input, if you have any ideas. </div><div><br></div><div>Anything that we can get into Theme-Check can only help developers and reviewers alike; however, some things simply require a human review. Theme Check can check for, say, pairing of add_theme_support( 'post-thumbnails' ) and calls to the_post_thumbnail(), but it can't ensure that the_post_thumbnail() is implemented correctly. Theme Check can ensure that License/License URI header tags are included in style.css, but it can't verify that the declared license is GPL-compatible, or that the supplied URI is valid. Theme Check can ensure that appropriate template tags are included, but it can't really duplicate the thoroughness of a human eye reviewing the Theme as-installed, and going through the Theme Unit Tests.</div>
<div><br></div><div>Assuming that the vast majority of Themes are submitted by developers who fully desire and intend to adhere to the guidelines (I am an optimist, after all), the best thing we can do for such developers is to provide a complete, thorough review of their initial submission. But, we must balance that against the expediency of the reviews. </div>
<div><br></div><div>Chip</div><div><br><div class="gmail_quote">On Tue, Oct 4, 2011 at 11:54 AM, Kirk Wight <span dir="ltr"><<a href="mailto:kwight@kwight.ca" target="_blank">kwight@kwight.ca</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Chip, I don't know you, but I already love you. WordPress loves you :)<div><br></div><div>I did a quick check of recent tickets closed-not-approved, and it seems a lot of themes are rejected for not meeting basic requirements, and/or best practices. I'm sure it's been mentioned before, but cutting off more of these themes at the gate (the upload point) would avoid a lot of wasted time on reviewers' part, and reduce the number of tickets per approved theme.</div>
<div><br></div><div>How does the upload check work, is it just a variation on the Theme Check plugin? Is there any way I/we can see what goes on there (SVN link maybe)? I'm not much of a PHP programmer, but if I can be of help to the person maintaining it by testing, etc, let me know.</div>
<div><br></div><div>In the meantime, I'll review themes :)</div><div><br></div><font color="#888888"><div>Kirk</div></font><div><div></div><div><div><br></div><div><br><br><div class="gmail_quote">On 4 October 2011 11:58, Chip Bennett <span dir="ltr"><<a href="mailto:chip@chipbennett.net" target="_blank">chip@chipbennett.net</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">The number of incoming tickets per month continues to climb steadily, but slowly. We are up to approximately 12 tickets per day:<div>
<img alt="oimg?key=0AmhRfB-XJH5_dE5FU0tvcTF2MmZwaWVkSV9PWVVHbFE&oid=1&zx=96ihgzsxiv84" title="oimg?key=0AmhRfB-XJH5_dE5FU0tvcTF2MmZwaWVkSV9PWVVHbFE&oid=1&zx=96ihgzsxiv84"><br>
And here is the monthly workload trend:</div><div><img alt="oimg?key=0AmhRfB-XJH5_dE5FU0tvcTF2MmZwaWVkSV9PWVVHbFE&oid=15&zx=am4vt98acw9k" title="oimg?key=0AmhRfB-XJH5_dE5FU0tvcTF2MmZwaWVkSV9PWVVHbFE&oid=15&zx=am4vt98acw9k"><br>
</div><div>As you can see, the total number of reviewers per month is also trending upward (note the spike, around the time that Justin Tadlock published his blog post calling for help with Theme reviews); however, note the high standard deviation, which in this case is indicative of a small number of reviewers performing the bulk of the reviews in any given month.</div>
<div><br></div><div>We can keep up with the incoming workload, if we average about 13 tickets closed per day.We're currently averaging 15 active Reviewers per month, which would require an average of one ticket per day, per active Reviewer.</div>
<div><br></div><div>To improve that number, we either need to get each active Reviewer (the bulk of whom review 5 or less tickets per month) to review considerably more tickets per month, or else we need to get more Reviewers active per month.</div>
<div><br></div><div>We have enough total Reviewers now that if everyone who has ever reviewed a ticket would review one ticket per week, we would keep up with the incoming workflow.</div><div><br></div><div>My ultimate goal is to improve the percentage of approved tickets, and a reduced number of tickets per Theme, which will drive down our total workload. As you can see, this has been difficult:</div>
<div><img alt="oimg?key=0AmhRfB-XJH5_dE5FU0tvcTF2MmZwaWVkSV9PWVVHbFE&oid=16&zx=z105kgyojdh6" title="oimg?key=0AmhRfB-XJH5_dE5FU0tvcTF2MmZwaWVkSV9PWVVHbFE&oid=16&zx=z105kgyojdh6"><br>
</div><div>We generally hold steady at about 20-25% of tickets (not including closed-newer-version tickets) being resolved as approved - though this past month we did have a nice up-tick. I don't currently have a reliable means to measure total number of tickets required per approval, although I would really like to track this metric.</div>
<div><br></div><div>So, that's where we are right now. As always, we're open to any ideas for improvement!</div><div><br></div><font color="#888888"><div>Chip</div></font><div><br><div class="gmail_quote"><div>
On Tue, Oct 4, 2011 at 9:39 AM, Kirk Wight <span dir="ltr"><<a href="mailto:kwight@kwight.ca" target="_blank">kwight@kwight.ca</a>></span> wrote:<br>
</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div>That's a great idea - keeping that #2 queue as empty as possible would be a huge boost in consistency, from a users point of view.<div>
<br></div><div>Out of curiosity, have you noticed any trends regarding increased submissions/workload? Maybe times of the month or year, or right after a WordPress point bump?</div>
<div><br></div><div><br></div>
<br></div><div>_______________________________________________<br>
theme-reviewers mailing list<br>
<a href="mailto:theme-reviewers@lists.wordpress.org" target="_blank">theme-reviewers@lists.wordpress.org</a><br>
<a href="http://lists.wordpress.org/mailman/listinfo/theme-reviewers" target="_blank">http://lists.wordpress.org/mailman/listinfo/theme-reviewers</a><br>
<br></div></blockquote></div><br></div>
<br>_______________________________________________<br>
theme-reviewers mailing list<br>
<a href="mailto:theme-reviewers@lists.wordpress.org" target="_blank">theme-reviewers@lists.wordpress.org</a><br>
<a href="http://lists.wordpress.org/mailman/listinfo/theme-reviewers" target="_blank">http://lists.wordpress.org/mailman/listinfo/theme-reviewers</a><br>
<br></blockquote></div><br></div>
</div></div><br>_______________________________________________<br>
theme-reviewers mailing list<br>
<a href="mailto:theme-reviewers@lists.wordpress.org" target="_blank">theme-reviewers@lists.wordpress.org</a><br>
<a href="http://lists.wordpress.org/mailman/listinfo/theme-reviewers" target="_blank">http://lists.wordpress.org/mailman/listinfo/theme-reviewers</a><br>
<br></blockquote></div><br></div>
_______________________________________________<br>theme-reviewers mailing list<br><a href="mailto:theme-reviewers@lists.wordpress.org" target="_blank">theme-reviewers@lists.wordpress.org</a><br><a href="http://lists.wordpress.org/mailman/listinfo/theme-reviewers" target="_blank">http://lists.wordpress.org/mailman/listinfo/theme-reviewers</a><br>
</blockquote></div><br></div></div></div></div><br>_______________________________________________<br>
theme-reviewers mailing list<br>
<a href="mailto:theme-reviewers@lists.wordpress.org">theme-reviewers@lists.wordpress.org</a><br>
<a href="http://lists.wordpress.org/mailman/listinfo/theme-reviewers" target="_blank">http://lists.wordpress.org/mailman/listinfo/theme-reviewers</a><br>
<br></blockquote></div><br></div>