[theme-reviewers] Goals and Process

Fri Jun 11 13:44:05 UTC 2010

Hey Gav,

I think what you say makes perfect sense, and I've liked where the
discussion is headed. I too wondered about the validity of 5 star rating
system for each item and for a lot of items in the checklist it's either
black or white... either it works or it doesn't work. However, I think there
are also items that could use a 5-point rating system because they are a bit
more subjective. Such as "Are comments displayed correctly" and "Is the
theme widgetized as fully as possible?".

I think the organization of the review process could be done to extreme
(using a bug tracking system like Bugzilla which would be a very effective
tool for communication with reviewers and authors), but if you wanted a
really simple solution you could also use a Google Docs Spreadsheet with the
Form functionality. It would only take a short amount of a time to setup a
form for the entire checklist. You could use checkboxes for the
black-and-white items, and you could use a 5 point radio group for the more
subjective items. Along with a text field for the name of the theme and free
form comments at the bottom of each sub-section. With this solution you
could use one spreadsheet per theme, or have all theme reviews in a single
spreadsheet (they have some cool summarization views for the form data).
Just a thought... the communication channels would still remain unresolved
at that point.

Tim

On Fri, Jun 11, 2010 at 7:07 AM, Gavin Pearce <GavinP at tbs.uk.com> wrote:

> Agreed almost completely.
>
> Not convinced about 1 - 5 ratings mind, you start to open up the process
> to the whim of individual reviewers again.
>
> Checklist should check for items that are:
>
>        * Critical items - No go, instant fail..
>        * Less critical - You get a 'advisory'.
>
> If you're template has more than X amount of "advisories", it doesn't
> pass. (Similar to the format of the UK Driving Test).
>
> I guess you could also get a different amount of "advisory points"
> depending on the problem, but it starts getting overly complicated
> again. Personally I think keep it as simple as possible - this will
> result in a much better consistency of what quality of template we
> allow/disallow.
>
> If built around this model, you can start to add extra criteria to
> improve the general quality of all templates, e.g.:
>        - To get a V2 version of the same template into the system, the
> author first has to demonstrate they acted upon at least some of the
> advisories issued first time round.
>
>
> Thoughts?       :)
>
>
> Gav
> /gavinpearce.com
>
>
> -----Original Message-----
> From: theme-reviewers-bounces at lists.wordpress.org
> [mailto:theme-reviewers-bounces at lists.wordpress.org] On Behalf Of
> chip at chipbennett.net
> Sent: 11 June 2010 13:43
> To: theme-reviewers at lists.wordpress.org
> Subject: Re: [theme-reviewers] Goals and Process
>
> I concur.
>
> I'm thinking that Joseph (Joe?) is getting several people started on an
> initial theme review, in order to give us some better context for coming
> up with the more formalized process.
>
> That's what I'm doing with my first review, anyway: just going through
> the
> TDC and evaluating the theme against each of the checklist points.
>
> I think, once we've got some context by reviewing an initial theme,
> we'll
> better be able to evaluate what criteria are critical/required
> (GO/NO-GO),
> and which ones are more subjective (1-5 rating, or whatever) - as well
> as
> how each subjective criterion (or category) should be weighted.
>
> Of course, my first theme assignment is taking a bit (sorry, Joseph -
> the
> one you gave me threw about five errors right off the bat!), so it may
> not
> be until the evening or tomorrow morning until I can report back on it.
> In
> the meantime, I'm appreciating the opportunity to read everyone else's
> initial take on the review process.
>
> (As for getting the feedback to the theme authors themselves: I assume
> Joseph is handling that?)
>
> Chip
>
> > Hi David,
> >
> > I agree with your points. I was thinking something like a scorecard
> too.
> >
> > What I understood so far, is that we will go to check the theme
> against
> > the listed points from Theme_Development_Checklist, right? Is this our
> > only job?
> >
> > I would appreciate a clarify from Joseph too.
> >
> > Regards,
> > Daniel
> >
> >
> >> Hi Joseph (and everyone else),
> >>
> >> First, is it Joseph or Joe?
> >>
> >> Second, can you clarify what exactly the goals are of a theme review?
> >> What are we trying to accomplish? I realize we are trying to improve
> the
> >> quality and reliability of themes, but how is that going to work?
> >> You've assigned some theme reviews to participants here, but they are
> >> reporting "problems" back to us, not to the author to fix.  Is there
> a
> >> process I am not seeing?
> >>
> >> It seems to me, and I apologize if I am speaking out of turn, is that
> >> all of the information on
> >>
> http://codex.wordpress.org/Theme_Development_Checklist#Theme_Unit_Test
> >> could be turned into a scorecard.  It's already broken up into
> sections.
> >>  Give each bullet list a line item with a 1-5 rating or a y/n and set
> a
> >> comment section for free form comments at the end of each sub
> section.
> >> Then a theme submission could come in, be assigned to 3-5 of us, and
> we
> >> each complete a scorecard.  (heck portions of it could be automated.)
> >> Then send these scorecards back to the theme author where they can
> >> digest what they have done right, and what needs improvement.
> Without
> >> being too rigid, this process is crying for some formality of method.
> >> This also gives the theme author feedback, but not at the whim of one
> >> reviewer since some of the points (aesthetics in particular) are very
> >> subjective.
> >>
> >> Just a thought...
> >>
> >> -David _______________________________________________
> >> theme-reviewers mailing list
> >> theme-reviewers at lists.wordpress.org
> >> http://lists.wordpress.org/mailman/listinfo/theme-reviewers
> >
> > --
> > Daniel Zilli _ Production Executive _ Jilliz Ltd.
> >
> > Phone: +66 (0) 85 334 1224 | Website: http://www.jilliz.com
> > 288/43 BaanMai Puttabucha 36, Bangmod, Thungkru Bangkok 10140.
> >
> > _______________________________________________
> > theme-reviewers mailing list
> > theme-reviewers at lists.wordpress.org
> > http://lists.wordpress.org/mailman/listinfo/theme-reviewers
> >
>
>
> _______________________________________________
> theme-reviewers mailing list
> theme-reviewers at lists.wordpress.org
> http://lists.wordpress.org/mailman/listinfo/theme-reviewers
> _______________________________________________
> theme-reviewers mailing list
> theme-reviewers at lists.wordpress.org
> http://lists.wordpress.org/mailman/listinfo/theme-reviewers
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wordpress.org/mailman/private/theme-reviewers/attachments/20100611/0b3cd861/attachment.htm>