<div>The uploader script is a combination of the Theme Check Plugin checks, as well as some (not-so) "secret sauce" tests (mostly security-related). The uploader runs all of the Theme-Check checks, and prevents upload of any Theme that has a WARNING or REQUIRED issue flagged.</div>
<div><br></div>Actually, around the WordPress 3.1 release, we widened the moat considerably with the uploader, which previously only *reported* the Theme Check issues, but didn't act on them. We did see a momentary dip in submitted Themes, but the numbers crept right back up.<div>
<br></div><div>We've automated *just* about everything feasible, though I'm sure Pross would love your help and input, if you have any ideas. </div><div><br></div><div>Anything that we can get into Theme-Check can only help developers and reviewers alike; however, some things simply require a human review. Theme Check can check for, say, pairing of add_theme_support( 'post-thumbnails' ) and calls to the_post_thumbnail(), but it can't ensure that the_post_thumbnail() is implemented correctly. Theme Check can ensure that License/License URI header tags are included in style.css, but it can't verify that the declared license is GPL-compatible, or that the supplied URI is valid. Theme Check can ensure that appropriate template tags are included, but it can't really duplicate the thoroughness of a human eye reviewing the Theme as-installed, and going through the Theme Unit Tests.</div>
<div><br></div><div>Assuming that the vast majority of Themes are submitted by developers who fully desire and intend to adhere to the guidelines (I am an optimist, after all), the best thing we can do for such developers is to provide a complete, thorough review of their initial submission. But, we must balance that against the expediency of the reviews. </div>
<div><br></div><div>Chip</div><div><br><div class="gmail_quote">On Tue, Oct 4, 2011 at 11:54 AM, Kirk Wight <span dir="ltr"><<a href="mailto:kwight@kwight.ca">kwight@kwight.ca</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
Chip, I don't know you, but I already love you. WordPress loves you :)<div><br></div><div>I did a quick check of recent tickets closed-not-approved, and it seems a lot of themes are rejected for not meeting basic requirements, and/or best practices. I'm sure it's been mentioned before, but cutting off more of these themes at the gate (the upload point) would avoid a lot of wasted time on reviewers' part, and reduce the number of tickets per approved theme.</div>
<div><br></div><div>How does the upload check work, is it just a variation on the Theme Check plugin? Is there any way I/we can see what goes on there (SVN link maybe)? I'm not much of a PHP programmer, but if I can be of help to the person maintaining it by testing, etc, let me know.</div>
<div><br></div><div>In the meantime, I'll review themes :)</div><div><br></div><font color="#888888"><div>Kirk</div></font><div><div></div><div class="h5"><div><br></div><div><br><br><div class="gmail_quote">On 4 October 2011 11:58, Chip Bennett <span dir="ltr"><<a href="mailto:chip@chipbennett.net" target="_blank">chip@chipbennett.net</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">The number of incoming tickets per month continues to climb steadily, but slowly. We are up to approximately 12 tickets per day:<div>
<img alt="oimg?key=0AmhRfB-XJH5_dE5FU0tvcTF2MmZwaWVkSV9PWVVHbFE&oid=1&zx=96ihgzsxiv84" title="oimg?key=0AmhRfB-XJH5_dE5FU0tvcTF2MmZwaWVkSV9PWVVHbFE&oid=1&zx=96ihgzsxiv84"><br>
And here is the monthly workload trend:</div><div><img alt="oimg?key=0AmhRfB-XJH5_dE5FU0tvcTF2MmZwaWVkSV9PWVVHbFE&oid=15&zx=am4vt98acw9k" title="oimg?key=0AmhRfB-XJH5_dE5FU0tvcTF2MmZwaWVkSV9PWVVHbFE&oid=15&zx=am4vt98acw9k"><br>
</div><div>As you can see, the total number of reviewers per month is also trending upward (note the spike, around the time that Justin Tadlock published his blog post calling for help with Theme reviews); however, note the high standard deviation, which in this case is indicative of a small number of reviewers performing the bulk of the reviews in any given month.</div>
<div><br></div><div>We can keep up with the incoming workload, if we average about 13 tickets closed per day.We're currently averaging 15 active Reviewers per month, which would require an average of one ticket per day, per active Reviewer.</div>
<div><br></div><div>To improve that number, we either need to get each active Reviewer (the bulk of whom review 5 or less tickets per month) to review considerably more tickets per month, or else we need to get more Reviewers active per month.</div>
<div><br></div><div>We have enough total Reviewers now that if everyone who has ever reviewed a ticket would review one ticket per week, we would keep up with the incoming workflow.</div><div><br></div><div>My ultimate goal is to improve the percentage of approved tickets, and a reduced number of tickets per Theme, which will drive down our total workload. As you can see, this has been difficult:</div>
<div><img alt="oimg?key=0AmhRfB-XJH5_dE5FU0tvcTF2MmZwaWVkSV9PWVVHbFE&oid=16&zx=z105kgyojdh6" title="oimg?key=0AmhRfB-XJH5_dE5FU0tvcTF2MmZwaWVkSV9PWVVHbFE&oid=16&zx=z105kgyojdh6"><br>
</div><div>We generally hold steady at about 20-25% of tickets (not including closed-newer-version tickets) being resolved as approved - though this past month we did have a nice up-tick. I don't currently have a reliable means to measure total number of tickets required per approval, although I would really like to track this metric.</div>
<div><br></div><div>So, that's where we are right now. As always, we're open to any ideas for improvement!</div><div><br></div><font color="#888888"><div>Chip</div></font><div><br><div class="gmail_quote"><div>
On Tue, Oct 4, 2011 at 9:39 AM, Kirk Wight <span dir="ltr"><<a href="mailto:kwight@kwight.ca" target="_blank">kwight@kwight.ca</a>></span> wrote:<br>
</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div>That's a great idea - keeping that #2 queue as empty as possible would be a huge boost in consistency, from a users point of view.<div>
<br></div><div>Out of curiosity, have you noticed any trends regarding increased submissions/workload? Maybe times of the month or year, or right after a WordPress point bump?</div>
<div><br></div><div><br></div>
<br></div><div>_______________________________________________<br>
theme-reviewers mailing list<br>
<a href="mailto:theme-reviewers@lists.wordpress.org" target="_blank">theme-reviewers@lists.wordpress.org</a><br>
<a href="http://lists.wordpress.org/mailman/listinfo/theme-reviewers" target="_blank">http://lists.wordpress.org/mailman/listinfo/theme-reviewers</a><br>
<br></div></blockquote></div><br></div>
<br>_______________________________________________<br>
theme-reviewers mailing list<br>
<a href="mailto:theme-reviewers@lists.wordpress.org" target="_blank">theme-reviewers@lists.wordpress.org</a><br>
<a href="http://lists.wordpress.org/mailman/listinfo/theme-reviewers" target="_blank">http://lists.wordpress.org/mailman/listinfo/theme-reviewers</a><br>
<br></blockquote></div><br></div>
</div></div><br>_______________________________________________<br>
theme-reviewers mailing list<br>
<a href="mailto:theme-reviewers@lists.wordpress.org">theme-reviewers@lists.wordpress.org</a><br>
<a href="http://lists.wordpress.org/mailman/listinfo/theme-reviewers" target="_blank">http://lists.wordpress.org/mailman/listinfo/theme-reviewers</a><br>
<br></blockquote></div><br></div>