[theme-reviewers] Theme Review Feedback at WPTavern Forum

[Chip Bennett]

On Wed, Aug 25, 2010 at 10:49 AM, Otto <otto at ottodestruct.com> wrote:

> On Wed, Aug 25, 2010 at 7:27 AM, Chip Bennett <chip at chipbennett.net>
> wrote:
> > Disagree, 100%. The Theme repository is linked directly into the back-end
> of
> > WordPress, and represents and reflects WordPress.
> > Also, poor code quality and failure to keep Themes from becoming obsolete
> > through function deprecation and failure to include new/improved core
> > functions and features is *bad* for end users.
>
> Hey, no, don't get me wrong. We want good code. But that doesn't mean
> we have to punch people in the face until we get it.
>
> My main problem is with the "rejection" here. Nobody should ever be
> "rejected" by our community. That's an extremely bad way to deal with
> authors, especially when they need our help in learning how to improve
> their code.
>

Semantics. We're not "rejecting" a Theme (or its developer); we're
suggesting that a Theme *as submitted* not be approved for inclusion in the
repository. Absolutely nothing prevents the developer from fixing the Theme
and re-submitting, and we actively encourage them to do so.

>
> Here's my suggestions for how to adjust our process:
>
> Step 1: No rejections. Period. When you send a rejection to an author,
> well, that's a "final" state, isn't it? Where's the authors incentive
> to come back to us when we've told him that he's not welcome here?
>
> Step 2: No closing tickets until they're really done. You can't have a
> dialog when the ticket is closed, because nobody is listening at that
> point. Authors that have issues need to have a chance to respond to
> them and to be educated. Ideally, they'd be able to upload the changes
> directly using the ticket system somehow, instead of resubmitting the
> whole theme again. If we need tech improvements to fix this, so be it.
>

We have a very limited implementation of Trac to work with. A Theme is
uploaded using the uploader. Once the upload to SVN is complete, a Trac
ticket is auto-generated. The Theme developer doesn't have commit access to
the SVN directory for his Theme (nor do we reviewers). So, if he makes
improvements, his only recourse is to re-submit via the uploader - which by
necessity generates a new Trac ticket.

You'll have to ask Cais for exact details, but what you're asking for, I
believe, would require a LOT of manual intervention in order to make it
work.

If you can have some influence in improving our ability to take advantage of
Trac, then I think every one of us would be supportive of and grateful for
that improvement.

For example:

1) Find a way to link new revisions to the already open Ticket for a given
Theme
2) Add new options for ticket resolution, such as "needs further revision"
or whatever

Every one of us, I believe, would LOVE such improvements.

>
> Step 3: Improve the way we respond to problems. I understand that
> reviewing is difficult, but these cursory and short reviews I'm seeing
> in trac with a list of problems (many of which I do not consider to be
> "problems") is hard to parse, especially for a new author. Maybe we
> need a tool to help reviewers build a more verbose list, with links to
> the codex and longer explanations of why we think certain things
> should be included in themes. But simply saying "=> body_class is
> required" doesn't really seem to be particularly useful to a theme
> author who may not grasp this process.
>

We have to balance. I like to give very thorough reviews. But, if a Theme
has been submitted 2,3,4 or more times and *still* fails the review, then
IMO it is clear that the Theme developer isn't making the effort to ensure
that the Theme meets the Guidelines. We don't want to incentivize such
behavior.

>
> Basically, I want there to be a dialog between the reviewers and the
> authors, and to give the authors a chance and the tools necessary to
> improve. As it stands now, they really are not being given that
> chance. The numbers really do speak for themselves.
>

No, the numbers *don't* speak for themselves. The numbers don't give any
indication for *why* Themes are being not-accepted.

I think each of us is all for having the dialogue you want. But, again: Trac
is a mostly foreign concept to a lot of Theme developers. Not only are we
having to educate them regarding new review Guidelines, but we're also
having to educate on how to make use of the Trac system, and the tickets
generated for their Theme submissions.

>
> But in order for this to happen, I think we also need to use a little
> more common sense in some of these items. Yes, every theme should have
> body_class, but if a theme doesn't, well, it doesn't really hurt
> anything. So that should be a case where we tell the author "Hey, you
> should add body_class. Here's an explanation why." and then let them
> do that instead of simply saying "Oh ho, you failed our test! You're
> *rejected*."
>

If we had a system in place that would allow for revision/resubmission
within the same ticket, then I'd be all for using it. But we don't. So, we
either need to improve our system, or make the best of what we have. Perhaps
you can influence the former option; most of the rest of us can't, so we
must focus on the latter option.

As for helping Theme developers understand why certain criteria are
required: we've been working very hard on improving the Theme Review Codex
page, to make it more clear, understandable, and concise. We've added Codex
cross-references for darn-near everything. It is not too much to expect
Theme developers to keep abreast of the information available in the Codex,
especially regarding standard WordPress features, functions, tags, and
hooks. We're doing everything we can to point them in the right direction.

>
> This is what I'm really trying to get at here. The system as I've just
> experienced it is difficult, confusing, and leaves a bad taste in my
> mouth. I don't want to go through it again, to the point where I won't
> be submitting any more themes to the system but just releasing them
> myself instead. How does that help anybody, anywhere? How does that
> improve my skills as a theme author? How does that benefit the users
> of my theme, if they are having to upgrade manually instead of using
> the automatic upgrader? This poor review process I experienced has
> driven me away and seriously hurts everybody involved.
>

Have you actually "experienced" it? Have you reviewed a Theme? Have you
submitted a Theme for review?

>
> Can't we consider that and consider ways to fix it? Let's build a
> community here.
>
> > Go audit the 545 Themes that were rejected. You tell us what percentage
> of
> > those should have been approved, in your opinion, and why.
>
> All of them (maybe 98%) should have *eventually* been approved,
> instead of being outright rejected and the theme developers told that
> they were unwanted here. Instead, most of those people probably won't
> be coming back or fixing those themes. Because rejection didn't stop
> them from getting released, it just stopped them from getting released
> through WordPress.org and stopped them from having discussions with
> the people in our community about it.


Agreed, regarding eventual approval. If we can make that process easier for
both the Theme developers and the review team, then by all means, let's do
it.

But just because the *process* has issues, doesn't mean that the Guidelines
aren't sound.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wordpress.org/pipermail/theme-reviewers/attachments/20100825/85f02acd/attachment-0001.htm>