[theme-reviewers] Goals and Process

Fri Jun 11 16:28:28 UTC 2010

> 	* we shouldn't require/test that a Theme has all the "extra"
> features it could ever need;
> 	* if a theme *does* make use of a certain extra feature though,
> we should however test to make sure these extra features are implemented
> correctly.

Exactly!

> In regards to version support, I think this falls safely under
> "technical" and should be something we test, and not left to the end
> user to discover. The theme author can choose what versions they wish to
> support, and we can check that all functions used are present in that
> version of Wordpress. We are suddenly adding a big overhead by checking
> this though - but I can't think of a good way round it?

It's overhead that I wouldn't mind. In fact, I think it would be a
far-better use of our time and effort than critiquing aesthetics.

I think any newly submitted theme needs to support the latest version
(defined perhaps as the current release, or any current Release
Candidate?). Likewise for updated themes (if we are reviewing updates in
addition to new submissions).

Backwards compatibility should be defined by the author - though seeing
what functions/etc. are used (e.g. in conditionals) would also give us
clues. And if the author claims that a theme is backwards-compatible to
Version X, then we should ensure that claimed compatibility is true (e.g.
are calls functions implemented after Version X properly sanitized, with
conditionals and alternate output?).

Chip

> Gav
> //gavinpearce.com
>
>
> -----Original Message-----
> From: chip at chipbennett.net [mailto:chip at chipbennett.net]
> Sent: 11 June 2010 17:05
> To: Gavin Pearce
> Cc: theme-reviewers
> Subject: Re: [theme-reviewers] Goals and Process
>
> My concern with the pass/fail (or even, subjective rating) system for
> non-critical criteria is that we will be dictating things that don't
> need
> to be dictated.
>
> For example: widgetization. For a modern WordPress theme, I would agree
> that the theme needs to be widget-ready. (I wouldn't have argued the
> same
> thing a year or two ago, though.) But, the degree of widgetization
> depends
> on the theme itself. Perhaps it only makes sense to have a single,
> widgetized sidebar. Perhaps it make sense for the theme to be widgetized
> to within inches of its life.
>
> How do we rate those two themes one against the other? We don't really -
> nor do we really need to.
>
> But, ensuring that the layout and PHP are correct for whatever extent to
> which the theme is widgetized? That's critical. Ensuring that widget
> areas
> have meaningful descriptions, so that users know what goes where as they
> are populating them with widgets? That's pretty useful, too.
>
> For another example: options.
>
> Some themes are intended to be simple and elegant, with no need for an
> options page. (The one I will eventually release is intended to be a
> teaching tool, to help others learn how to use template tags and so on -
> with no options page.) Other themes are intended to be more complex, and
> need to make use of options pages. Again, these types of themes need to
> be
> treated differently by reviewers.
>
> That's why, while the TDC is a good starting point, it really needs to
> be
> revised/massaged to meet our specific needs as theme reviewers. I'm
> liking
> the discussion thus far to that end.
>
> Chip
>
>> Thanks for the feedback Tim, some great ideas with the how to format
> it.
>> I also agree with Devon earlier, though a PDF would be handy,
> somewhere
>> that responses can also be displayed to the original author would work
>> quicker.  A nice online app would be handy indeed.
>>
>>
>>
>> With the rating system though - I feel that also the subjective items
>> should be a case of just simply pass/fail - largely because by their
>> very nature, subjective items are somewhat subjective (excuse the
> pun).
>>
>>
>>
>> Is the theme widgetized as fully as possible?
>>
>> Are comments displayed correctly?
>>
>>
>>
>> On a scale of 1 - 5 would be very hard to judge consistently between
>> reviewers. What makes a widgetized theme a 4 rather than a 3 for
>> example? Also saying that, from past experience you'll also just end
> up
>> with a lot of 3's.
>>
>>
>>
>> If a template has a MAJOR problem, it fails the review instantly.
>>
>> If a template has a MINOR problem, we issue an advisory to the author
> to
>> make some changes, but continue to allow it.
>>
>> If a template has more than X advisories, it also fails.
>>
>>
>>
>> That way, a template that isn't fully widgetized will pass if that is
>> it's only problem, but a template that isn't fully widgetized, has
>> errors in the comment area, has formatting errors + others will fail.
>>
>>
>>
>> Simple yes/no checklist is the way forward  think, especially in terms
>> of simplicity, consistency and quality, for both the theme authors,
> and
>> ourseleves.
>>
>>
>>
>> Feel free to tell me I'm completely wrong though.  ;  )
>>
>>
>>
>> Gav
>>
>> /gavinpearce.com
>>
>>
>>
>>
>>
>>
>>
>> From: tgolen at gmail.com [mailto:tgolen at gmail.com] On Behalf Of Tim
> Golen
>> Sent: 11 June 2010 14:44
>> To: Gavin Pearce
>> Cc: theme-reviewers
>> Subject: Re: [theme-reviewers] Goals and Process
>>
>>
>>
>> Hey Gav,
>>
>>
>>
>> I think what you say makes perfect sense, and I've liked where the
>> discussion is headed. I too wondered about the validity of 5 star
> rating
>> system for each item and for a lot of items in the checklist it's
> either
>> black or white... either it works or it doesn't work. However, I think
>> there are also items that could use a 5-point rating system because
> they
>> are a bit more subjective. Such as "Are comments displayed correctly"
>> and "Is the theme widgetized as fully as possible?".
>>
>>
>>
>> I think the organization of the review process could be done to
> extreme
>> (using a bug tracking system like Bugzilla which would be a very
>> effective tool for communication with reviewers and authors), but if
> you
>> wanted a really simple solution you could also use a Google Docs
>> Spreadsheet with the Form functionality. It would only take a short
>> amount of a time to setup a form for the entire checklist. You could
> use
>> checkboxes for the black-and-white items, and you could use a 5 point
>> radio group for the more subjective items. Along with a text field for
>> the name of the theme and free form comments at the bottom of each
>> sub-section. With this solution you could use one spreadsheet per
> theme,
>> or have all theme reviews in a single spreadsheet (they have some cool
>> summarization views for the form data). Just a thought... the
>> communication channels would still remain unresolved at that point.
>>
>>
>>
>> Tim
>>
>> On Fri, Jun 11, 2010 at 7:07 AM, Gavin Pearce <GavinP at tbs.uk.com>
> wrote:
>>
>> Agreed almost completely.
>>
>> Not convinced about 1 - 5 ratings mind, you start to open up the
> process
>> to the whim of individual reviewers again.
>>
>> Checklist should check for items that are:
>>
>>        * Critical items - No go, instant fail..
>>        * Less critical - You get a 'advisory'.
>>
>> If you're template has more than X amount of "advisories", it doesn't
>> pass. (Similar to the format of the UK Driving Test).
>>
>> I guess you could also get a different amount of "advisory points"
>> depending on the problem, but it starts getting overly complicated
>> again. Personally I think keep it as simple as possible - this will
>> result in a much better consistency of what quality of template we
>> allow/disallow.
>>
>> If built around this model, you can start to add extra criteria to
>> improve the general quality of all templates, e.g.:
>>        - To get a V2 version of the same template into the system, the
>> author first has to demonstrate they acted upon at least some of the
>> advisories issued first time round.
>>
>>
>> Thoughts?       :)
>>
>>
>> Gav
>> /gavinpearce.com
>>
>>
>>
>> -----Original Message-----
>> From: theme-reviewers-bounces at lists.wordpress.org
>> [mailto:theme-reviewers-bounces at lists.wordpress.org] On Behalf Of
>> chip at chipbennett.net
>> Sent: 11 June 2010 13:43
>> To: theme-reviewers at lists.wordpress.org
>> Subject: Re: [theme-reviewers] Goals and Process
>>
>> I concur.
>>
>> I'm thinking that Joseph (Joe?) is getting several people started on
> an
>> initial theme review, in order to give us some better context for
> coming
>> up with the more formalized process.
>>
>> That's what I'm doing with my first review, anyway: just going through
>> the
>> TDC and evaluating the theme against each of the checklist points.
>>
>> I think, once we've got some context by reviewing an initial theme,
>> we'll
>> better be able to evaluate what criteria are critical/required
>> (GO/NO-GO),
>> and which ones are more subjective (1-5 rating, or whatever) - as well
>> as
>> how each subjective criterion (or category) should be weighted.
>>
>> Of course, my first theme assignment is taking a bit (sorry, Joseph -
>> the
>> one you gave me threw about five errors right off the bat!), so it may
>> not
>> be until the evening or tomorrow morning until I can report back on
> it.
>> In
>> the meantime, I'm appreciating the opportunity to read everyone else's
>> initial take on the review process.
>>
>> (As for getting the feedback to the theme authors themselves: I assume
>> Joseph is handling that?)
>>
>> Chip
>>
>>> Hi David,
>>>
>>> I agree with your points. I was thinking something like a scorecard
>> too.
>>>
>>> What I understood so far, is that we will go to check the theme
>> against
>>> the listed points from Theme_Development_Checklist, right? Is this
> our
>>> only job?
>>>
>>> I would appreciate a clarify from Joseph too.
>>>
>>> Regards,
>>> Daniel
>>>
>>>
>>>> Hi Joseph (and everyone else),
>>>>
>>>> First, is it Joseph or Joe?
>>>>
>>>> Second, can you clarify what exactly the goals are of a theme
> review?
>>>> What are we trying to accomplish? I realize we are trying to improve
>> the
>>>> quality and reliability of themes, but how is that going to work?
>>>> You've assigned some theme reviews to participants here, but they
> are
>>>> reporting "problems" back to us, not to the author to fix.  Is there
>> a
>>>> process I am not seeing?
>>>>
>>>> It seems to me, and I apologize if I am speaking out of turn, is
> that
>>>> all of the information on
>>>>
>> http://codex.wordpress.org/Theme_Development_Checklist#Theme_Unit_Test
>>>> could be turned into a scorecard.  It's already broken up into
>> sections.
>>>>  Give each bullet list a line item with a 1-5 rating or a y/n and
> set
>> a
>>>> comment section for free form comments at the end of each sub
>> section.
>>>> Then a theme submission could come in, be assigned to 3-5 of us, and
>> we
>>>> each complete a scorecard.  (heck portions of it could be
> automated.)
>>>> Then send these scorecards back to the theme author where they can
>>>> digest what they have done right, and what needs improvement.
>> Without
>>>> being too rigid, this process is crying for some formality of
> method.
>>>> This also gives the theme author feedback, but not at the whim of
> one
>>>> reviewer since some of the points (aesthetics in particular) are
> very
>>>> subjective.
>>>>
>>>> Just a thought...
>>>>
>>>> -David _______________________________________________
>>>> theme-reviewers mailing list
>>>> theme-reviewers at lists.wordpress.org
>>>> http://lists.wordpress.org/mailman/listinfo/theme-reviewers
>>>
>>> --
>>> Daniel Zilli _ Production Executive _ Jilliz Ltd.
>>>
>>> Phone: +66 (0) 85 334 1224 | Website: http://www.jilliz.com
>>> 288/43 BaanMai Puttabucha 36, Bangmod, Thungkru Bangkok 10140.
>>>
>>> _______________________________________________
>>> theme-reviewers mailing list
>>> theme-reviewers at lists.wordpress.org
>>> http://lists.wordpress.org/mailman/listinfo/theme-reviewers
>>>
>>
>>
>> _______________________________________________
>> theme-reviewers mailing list
>> theme-reviewers at lists.wordpress.org
>> http://lists.wordpress.org/mailman/listinfo/theme-reviewers
>> _______________________________________________
>> theme-reviewers mailing list
>> theme-reviewers at lists.wordpress.org
>> http://lists.wordpress.org/mailman/listinfo/theme-reviewers
>>
>>
>>
>> _______________________________________________
>> theme-reviewers mailing list
>> theme-reviewers at lists.wordpress.org
>> http://lists.wordpress.org/mailman/listinfo/theme-reviewers
>>
>
>
> _______________________________________________
> theme-reviewers mailing list
> theme-reviewers at lists.wordpress.org
> http://lists.wordpress.org/mailman/listinfo/theme-reviewers
>