[theme-reviewers] Goals and Process

Fri Jun 11 16:16:08 UTC 2010

> Hi David,
>
>
>
> Without trying to sound too awkward, I'm afraid I still disagree.
>
> *	The theme has a clear visual hierarchy.   - It's either clear, or it's
> not. If it's "almost" clear but not completely clear, then it's "not
> clear" and would therefore be a no + advisory to improve slightly if
> possible.

And how do we define "clear visual hierarchy". If we use the "I'll know it
when I see it" approach, then the standard will be subjective, and
unevenly evaluated. So, if we use this approach, we need an objective
definition of "clear visual hierarchy."

> *	The navigation is easy to understand.   - It either easy to understand,
> or it isn't. If it's "almost" easy to understand, but not completely easy
> to understand, then it's a no and as above.

Define "easy to understand navigation"? Easy for whom? How "easy" is easy
enough?

> *	Where possible, decorative images are in CSS.    - All decorative images
> are in the CSS .. if they are then yes, if not no + another advisory.
> *	The theme uses CSS for all presentational aspects.  - It either uses CSS
> for all presentation aspects, or it doesn't.

Again: we need a standard defined, if this is to be a yes/no criterion.

> Again, remember an advisory would be only for things that are minor, and
> you'd have to get a few of them to fail.
>
> As for things that are subjective on the eye of the reviewer, such as your
> example - a "is that hierarchy 100% clear (yes/no)", is a lot easier to
> resolve with a second pair of eyes than "is that hierarchy a 2 or a 3?".

Unless we have objective definitions for what constitutes a 1, 2, 3, 4, or
5 with respect to "clear visual hierarchy".

> I also feel, that while 'rating' things would be nice, we don't gain
> anything extra by doing it.
>
> E.g.
>
> Â·         The theme uses CSS for all presentational aspects.
>
> No matter whether we rated this as a 2, 3 or 4, we'd still have the same
> bit of advice to tell the author - "Put more of your presentational images
> into the CSS".
>
> Gav
> //gavinpearce.com

Again, we need clear definitions. Should this criterion downgrade, say, a
table that is used to control layout rather than to render tabular data?
What all "presentational aspects" should be evaluated? (More for theme
authors than for us, I suppose.)

So, ultimately, I'm all for explicit, yes/no criteria - but they have to
be clearly, precisely, and objectively defined. That could be a tall order
for some of our criteria.

Chip

> From: david at mirmillo.com [mailto:david at mirmillo.com] On Behalf Of David
> Damstra
> Sent: 11 June 2010 15:25
> To: Gavin Pearce
> Cc: theme-reviewers
> Subject: Re: [theme-reviewers] Goals and Process
>
>
>
> Gavin, I completely agree about the Y/N flags... but there are several
> items on the checklist that are truly subjective, for example:
>
> *	The theme has a clear visual hierarchy.
> *	The navigation is easy to understand.
>
> *	Where possible, decorative images are in CSS.
> *	The theme uses CSS for all presentational aspects.
>
> Not to mention general aesthetic appeals and eye of the beholder stuff.
> That is where I think your need rating scales.  Ranging from not at all
> (1) to For the most part(5) and address advisories in the free form
> comments.  I also think this is where it actually is good to have more
> than one reviewer per theme, so that many eyes can see different issues,
> and clear visual hierarchy can be interpreted different ways.  This also
> keeps creative license in the theme author's hands.  Maybe they are
> intending to create a dark navigation that I perceive as "not clear"
> because my monitor is horribly calibrated.
>
> Low subjective ratings without comments are worthless, use the comments to
> explain the rating and recommend a course of action and constructive
> criticism.  But the subjective portions are advice for improvement.
> Whereas the Y/N portions of the site are more make it or break it.  If you
> don't pass the critical Y/N portions (or at least a minimum of them) then
> the theme is no fly until they are fixed.
>
> Again, I am just talking here... I am still unclear on what the expected
> outcome of these theme reviews are. Hoping to hear some clarification from
> Joseph.
>
>
>
>
> On Fri, Jun 11, 2010 at 10:11 AM, Gavin Pearce <GavinP at tbs.uk.com> wrote:
>
> Thanks for the feedback Tim, some great ideas with the how to format it. I
> also agree with Devon earlier, though a PDF would be handy, somewhere that
> responses can also be displayed to the original author would work quicker.
>  A nice online app would be handy indeed.
>
>
>
> With the rating system though - I feel that also the subjective items
> should be a case of just simply pass/fail - largely because by their very
> nature, subjective items are somewhat subjective (excuse the pun).
>
>
>
> Is the theme widgetized as fully as possible?
>
> Are comments displayed correctly?
>
>
>
> On a scale of 1 - 5 would be very hard to judge consistently between
> reviewers. What makes a widgetized theme a 4 rather than a 3 for example?
> Also saying that, from past experience you'll also just end up with a lot
> of 3's.
>
>
>
> If a template has a MAJOR problem, it fails the review instantly.
>
> If a template has a MINOR problem, we issue an advisory to the author to
> make some changes, but continue to allow it.
>
> If a template has more than X advisories, it also fails.
>
>
>
> That way, a template that isn't fully widgetized will pass if that is it's
> only problem, but a template that isn't fully widgetized, has errors in
> the comment area, has formatting errors + others will fail.
>
>
>
> Simple yes/no checklist is the way forward  think, especially in terms of
> simplicity, consistency and quality, for both the theme authors, and
> ourseleves.
>
>
>
> Feel free to tell me I'm completely wrong though.  ;  )
>
>
>
> Gav
>
> /gavinpearce.com
>
>
>
>
>
>
>
> From: tgolen at gmail.com [mailto:tgolen at gmail.com] On Behalf Of Tim Golen
> Sent: 11 June 2010 14:44
> To: Gavin Pearce
> Cc: theme-reviewers
>
>
> Subject: Re: [theme-reviewers] Goals and Process
>
>
>
> Hey Gav,
>
>
>
> I think what you say makes perfect sense, and I've liked where the
> discussion is headed. I too wondered about the validity of 5 star rating
> system for each item and for a lot of items in the checklist it's either
> black or white... either it works or it doesn't work. However, I think
> there are also items that could use a 5-point rating system because they
> are a bit more subjective. Such as "Are comments displayed correctly" and
> "Is the theme widgetized as fully as possible?".
>
>
>
> I think the organization of the review process could be done to extreme
> (using a bug tracking system like Bugzilla which would be a very effective
> tool for communication with reviewers and authors), but if you wanted a
> really simple solution you could also use a Google Docs Spreadsheet with
> the Form functionality. It would only take a short amount of a time to
> setup a form for the entire checklist. You could use checkboxes for the
> black-and-white items, and you could use a 5 point radio group for the
> more subjective items. Along with a text field for the name of the theme
> and free form comments at the bottom of each sub-section. With this
> solution you could use one spreadsheet per theme, or have all theme
> reviews in a single spreadsheet (they have some cool summarization views
> for the form data). Just a thought... the communication channels would
> still remain unresolved at that point.
>
>
>
> Tim
>
> On Fri, Jun 11, 2010 at 7:07 AM, Gavin Pearce <GavinP at tbs.uk.com> wrote:
>
> Agreed almost completely.
>
> Not convinced about 1 - 5 ratings mind, you start to open up the process
> to the whim of individual reviewers again.
>
> Checklist should check for items that are:
>
>        * Critical items - No go, instant fail..
>        * Less critical - You get a 'advisory'.
>
> If you're template has more than X amount of "advisories", it doesn't
> pass. (Similar to the format of the UK Driving Test).
>
> I guess you could also get a different amount of "advisory points"
> depending on the problem, but it starts getting overly complicated
> again. Personally I think keep it as simple as possible - this will
> result in a much better consistency of what quality of template we
> allow/disallow.
>
> If built around this model, you can start to add extra criteria to
> improve the general quality of all templates, e.g.:
>        - To get a V2 version of the same template into the system, the
> author first has to demonstrate they acted upon at least some of the
> advisories issued first time round.
>
>
> Thoughts?       :)
>
>
> Gav
> /gavinpearce.com
>
>
>
> -----Original Message-----
> From: theme-reviewers-bounces at lists.wordpress.org
> [mailto:theme-reviewers-bounces at lists.wordpress.org] On Behalf Of
> chip at chipbennett.net
> Sent: 11 June 2010 13:43
> To: theme-reviewers at lists.wordpress.org
> Subject: Re: [theme-reviewers] Goals and Process
>
> I concur.
>
> I'm thinking that Joseph (Joe?) is getting several people started on an
> initial theme review, in order to give us some better context for coming
> up with the more formalized process.
>
> That's what I'm doing with my first review, anyway: just going through
> the
> TDC and evaluating the theme against each of the checklist points.
>
> I think, once we've got some context by reviewing an initial theme,
> we'll
> better be able to evaluate what criteria are critical/required
> (GO/NO-GO),
> and which ones are more subjective (1-5 rating, or whatever) - as well
> as
> how each subjective criterion (or category) should be weighted.
>
> Of course, my first theme assignment is taking a bit (sorry, Joseph -
> the
> one you gave me threw about five errors right off the bat!), so it may
> not
> be until the evening or tomorrow morning until I can report back on it.
> In
> the meantime, I'm appreciating the opportunity to read everyone else's
> initial take on the review process.
>
> (As for getting the feedback to the theme authors themselves: I assume
> Joseph is handling that?)
>
> Chip
>
>> Hi David,
>>
>> I agree with your points. I was thinking something like a scorecard
> too.
>>
>> What I understood so far, is that we will go to check the theme
> against
>> the listed points from Theme_Development_Checklist, right? Is this our
>> only job?
>>
>> I would appreciate a clarify from Joseph too.
>>
>> Regards,
>> Daniel
>>
>>
>>> Hi Joseph (and everyone else),
>>>
>>> First, is it Joseph or Joe?
>>>
>>> Second, can you clarify what exactly the goals are of a theme review?
>>> What are we trying to accomplish? I realize we are trying to improve
> the
>>> quality and reliability of themes, but how is that going to work?
>>> You've assigned some theme reviews to participants here, but they are
>>> reporting "problems" back to us, not to the author to fix.  Is there
> a
>>> process I am not seeing?
>>>
>>> It seems to me, and I apologize if I am speaking out of turn, is that
>>> all of the information on
>>>
> http://codex.wordpress.org/Theme_Development_Checklist#Theme_Unit_Test
>>> could be turned into a scorecard.  It's already broken up into
> sections.
>>>  Give each bullet list a line item with a 1-5 rating or a y/n and set
> a
>>> comment section for free form comments at the end of each sub
> section.
>>> Then a theme submission could come in, be assigned to 3-5 of us, and
> we
>>> each complete a scorecard.  (heck portions of it could be automated.)
>>> Then send these scorecards back to the theme author where they can
>>> digest what they have done right, and what needs improvement.
> Without
>>> being too rigid, this process is crying for some formality of method.
>>> This also gives the theme author feedback, but not at the whim of one
>>> reviewer since some of the points (aesthetics in particular) are very
>>> subjective.
>>>
>>> Just a thought...
>>>
>>> -David _______________________________________________
>>> theme-reviewers mailing list
>>> theme-reviewers at lists.wordpress.org
>>> http://lists.wordpress.org/mailman/listinfo/theme-reviewers
>>
>> --
>> Daniel Zilli _ Production Executive _ Jilliz Ltd.
>>
>> Phone: +66 (0) 85 334 1224 | Website: http://www.jilliz.com
>> 288/43 BaanMai Puttabucha 36, Bangmod, Thungkru Bangkok 10140.
>>
>> _______________________________________________
>> theme-reviewers mailing list
>> theme-reviewers at lists.wordpress.org
>> http://lists.wordpress.org/mailman/listinfo/theme-reviewers
>>
>
>
> _______________________________________________
> theme-reviewers mailing list
> theme-reviewers at lists.wordpress.org
> http://lists.wordpress.org/mailman/listinfo/theme-reviewers
> _______________________________________________
> theme-reviewers mailing list
> theme-reviewers at lists.wordpress.org
> http://lists.wordpress.org/mailman/listinfo/theme-reviewers
>
>
>
>
> _______________________________________________
> theme-reviewers mailing list
> theme-reviewers at lists.wordpress.org
> http://lists.wordpress.org/mailman/listinfo/theme-reviewers
>
>
>
> _______________________________________________
> theme-reviewers mailing list
> theme-reviewers at lists.wordpress.org
> http://lists.wordpress.org/mailman/listinfo/theme-reviewers
>