[theme-reviewers] theme-reviewers Digest, Vol 1, Issue 32

Gavin Pearce GavinP at tbs.uk.com
Fri Jun 11 17:47:24 UTC 2010 

Take the list there with a pinch of salt though - there's a few tiny errors I've noticed that are not 'quite' as per the WP Codex doc.

Gav
//gavinpearce.com

-----Original Message-----
From: theme-reviewers-bounces at lists.wordpress.org [mailto:theme-reviewers-bounces at lists.wordpress.org] On Behalf Of Johan Dahlstrom
Sent: 11 June 2010 18:40
Cc: theme-reviewers at lists.wordpress.org
Subject: Re: [theme-reviewers] theme-reviewers Digest, Vol 1, Issue 32

Ohhh.. Add that to a P2 and we're rockin'!

/J

On 11 June 2010 17:12, Edward Caissie <edward.caissie at gmail.com> wrote:
> I thought I had come across something of an interactive checklist a while
> back, it just took me some time to find it in my bookmarks ... have a look
> at this: http://www.wplover.com/lab/theme-development-checklist
>
> Might be a place to start from ...
>
>
> Cais
>
> On Fri, Jun 11, 2010 at 10:55 AM,
> <theme-reviewers-request at lists.wordpress.org> wrote:
>>
>> Send theme-reviewers mailing list submissions to
>>        theme-reviewers at lists.wordpress.org
>>
>> To subscribe or unsubscribe via the World Wide Web, visit
>>        http://lists.wordpress.org/mailman/listinfo/theme-reviewers
>> or, via email, send a message with subject or body 'help' to
>>        theme-reviewers-request at lists.wordpress.org
>>
>> You can reach the person managing the list at
>>        theme-reviewers-owner at lists.wordpress.org
>>
>> When replying, please edit your Subject line so it is more specific
>> than "Re: Contents of theme-reviewers digest..."
>>
>>
>> Today's Topics:
>>
>>   1. Re: Goals and Process (Gavin Pearce)
>>
>>
>> ----------------------------------------------------------------------
>>
>> Message: 1
>> Date: Fri, 11 Jun 2010 15:43:01 +0100
>> From: "Gavin Pearce" <GavinP at tbs.uk.com>
>> Subject: Re: [theme-reviewers] Goals and Process
>> To: "theme-reviewers" <theme-reviewers at lists.wordpress.org>
>> Message-ID:
>>        <CD47955B7E065D48AB5CF226C2ED75C0295BFE at tbs-sbs01.TBS.local>
>> Content-Type: text/plain; charset="utf-8"
>>
>> Hi David,
>>
>>
>>
>> Without trying to sound too awkward, I'm afraid I still disagree.
>>
>> *       The theme has a clear visual hierarchy.   - It's either clear, or
>> it's not. If it's "almost" clear but not completely clear, then it's "not
>> clear" and would therefore be a no + advisory to improve slightly if
>> possible.
>> *       The navigation is easy to understand.   - It either easy to
>> understand, or it isn't. If it's "almost" easy to understand, but not
>> completely easy to understand, then it's a no and as above.
>>
>> *       Where possible, decorative images are in CSS.    - All decorative
>> images are in the CSS .. if they are then yes, if not no + another advisory.
>> *       The theme uses CSS for all presentational aspects.  - It either
>> uses CSS for all presentation aspects, or it doesn't.
>>
>> Again, remember an advisory would be only for things that are minor, and
>> you'd have to get a few of them to fail.
>>
>> As for things that are subjective on the eye of the reviewer, such as your
>> example - a "is that hierarchy 100% clear (yes/no)", is a lot easier to
>> resolve with a second pair of eyes than "is that hierarchy a 2 or a 3?".
>>
>> I also feel, that while 'rating' things would be nice, we don't gain
>> anything extra by doing it.
>>
>> E.g.
>>
>> ?         The theme uses CSS for all presentational aspects.
>>
>> No matter whether we rated this as a 2, 3 or 4, we'd still have the same
>> bit of advice to tell the author - "Put more of your presentational images
>> into the CSS".
>>
>> Gav
>> //gavinpearce.com
>>
>>
>>
>> From: david at mirmillo.com [mailto:david at mirmillo.com] On Behalf Of David
>> Damstra
>> Sent: 11 June 2010 15:25
>> To: Gavin Pearce
>> Cc: theme-reviewers
>> Subject: Re: [theme-reviewers] Goals and Process
>>
>>
>>
>> Gavin, I completely agree about the Y/N flags... but there are several
>> items on the checklist that are truly subjective, for example:
>>
>> *       The theme has a clear visual hierarchy.
>> *       The navigation is easy to understand.
>>
>> *       Where possible, decorative images are in CSS.
>> *       The theme uses CSS for all presentational aspects.
>>
>> Not to mention general aesthetic appeals and eye of the beholder stuff.
>>  That is where I think your need rating scales.  Ranging from not at all (1)
>> to For the most part(5) and address advisories in the free form comments.  I
>> also think this is where it actually is good to have more than one reviewer
>> per theme, so that many eyes can see different issues, and clear visual
>> hierarchy can be interpreted different ways.  This also keeps creative
>> license in the theme author's hands.  Maybe they are intending to create a
>> dark navigation that I perceive as "not clear" because my monitor is
>> horribly calibrated.
>>
>> Low subjective ratings without comments are worthless, use the comments to
>> explain the rating and recommend a course of action and constructive
>> criticism.  But the subjective portions are advice for improvement.  Whereas
>> the Y/N portions of the site are more make it or break it.  If you don't
>> pass the critical Y/N portions (or at least a minimum of them) then the
>> theme is no fly until they are fixed.
>>
>> Again, I am just talking here... I am still unclear on what the expected
>> outcome of these theme reviews are. Hoping to hear some clarification from
>> Joseph.
>>
>>
>>
>>
>> On Fri, Jun 11, 2010 at 10:11 AM, Gavin Pearce <GavinP at tbs.uk.com> wrote:
>>
>> Thanks for the feedback Tim, some great ideas with the how to format it. I
>> also agree with Devon earlier, though a PDF would be handy, somewhere that
>> responses can also be displayed to the original author would work quicker.
>>  A nice online app would be handy indeed.
>>
>>
>>
>> With the rating system though - I feel that also the subjective items
>> should be a case of just simply pass/fail - largely because by their very
>> nature, subjective items are somewhat subjective (excuse the pun).
>>
>>
>>
>> Is the theme widgetized as fully as possible?
>>
>> Are comments displayed correctly?
>>
>>
>>
>> On a scale of 1 - 5 would be very hard to judge consistently between
>> reviewers. What makes a widgetized theme a 4 rather than a 3 for example?
>> Also saying that, from past experience you'll also just end up with a lot of
>> 3's.
>>
>>
>>
>> If a template has a MAJOR problem, it fails the review instantly.
>>
>> If a template has a MINOR problem, we issue an advisory to the author to
>> make some changes, but continue to allow it.
>>
>> If a template has more than X advisories, it also fails.
>>
>>
>>
>> That way, a template that isn't fully widgetized will pass if that is it's
>> only problem, but a template that isn't fully widgetized, has errors in the
>> comment area, has formatting errors + others will fail.
>>
>>
>>
>> Simple yes/no checklist is the way forward  think, especially in terms of
>> simplicity, consistency and quality, for both the theme authors, and
>> ourseleves.
>>
>>
>>
>> Feel free to tell me I'm completely wrong though.  ;  )
>>
>>
>>
>> Gav
>>
>> /gavinpearce.com
>>
>>
>>
>>
>>
>>
>>
>> From: tgolen at gmail.com [mailto:tgolen at gmail.com] On Behalf Of Tim Golen
>> Sent: 11 June 2010 14:44
>> To: Gavin Pearce
>> Cc: theme-reviewers
>>
>>
>> Subject: Re: [theme-reviewers] Goals and Process
>>
>>
>>
>> Hey Gav,
>>
>>
>>
>> I think what you say makes perfect sense, and I've liked where the
>> discussion is headed. I too wondered about the validity of 5 star rating
>> system for each item and for a lot of items in the checklist it's either
>> black or white... either it works or it doesn't work. However, I think there
>> are also items that could use a 5-point rating system because they are a bit
>> more subjective. Such as "Are comments displayed correctly" and "Is the
>> theme widgetized as fully as possible?".
>>
>>
>>
>> I think the organization of the review process could be done to extreme
>> (using a bug tracking system like Bugzilla which would be a very effective
>> tool for communication with reviewers and authors), but if you wanted a
>> really simple solution you could also use a Google Docs Spreadsheet with the
>> Form functionality. It would only take a short amount of a time to setup a
>> form for the entire checklist. You could use checkboxes for the
>> black-and-white items, and you could use a 5 point radio group for the more
>> subjective items. Along with a text field for the name of the theme and free
>> form comments at the bottom of each sub-section. With this solution you
>> could use one spreadsheet per theme, or have all theme reviews in a single
>> spreadsheet (they have some cool summarization views for the form data).
>> Just a thought... the communication channels would still remain unresolved
>> at that point.
>>
>>
>>
>> Tim
>>
>> On Fri, Jun 11, 2010 at 7:07 AM, Gavin Pearce <GavinP at tbs.uk.com> wrote:
>>
>> Agreed almost completely.
>>
>> Not convinced about 1 - 5 ratings mind, you start to open up the process
>> to the whim of individual reviewers again.
>>
>> Checklist should check for items that are:
>>
>>       * Critical items - No go, instant fail..
>>       * Less critical - You get a 'advisory'.
>>
>> If you're template has more than X amount of "advisories", it doesn't
>> pass. (Similar to the format of the UK Driving Test).
>>
>> I guess you could also get a different amount of "advisory points"
>> depending on the problem, but it starts getting overly complicated
>> again. Personally I think keep it as simple as possible - this will
>> result in a much better consistency of what quality of template we
>> allow/disallow.
>>
>> If built around this model, you can start to add extra criteria to
>> improve the general quality of all templates, e.g.:
>>       - To get a V2 version of the same template into the system, the
>> author first has to demonstrate they acted upon at least some of the
>> advisories issued first time round.
>>
>>
>> Thoughts?       :)
>>
>>
>> Gav
>> /gavinpearce.com
>>
>>
>>
>> -----Original Message-----
>> From: theme-reviewers-bounces at lists.wordpress.org
>> [mailto:theme-reviewers-bounces at lists.wordpress.org] On Behalf Of
>> chip at chipbennett.net
>> Sent: 11 June 2010 13:43
>> To: theme-reviewers at lists.wordpress.org
>> Subject: Re: [theme-reviewers] Goals and Process
>>
>> I concur.
>>
>> I'm thinking that Joseph (Joe?) is getting several people started on an
>> initial theme review, in order to give us some better context for coming
>> up with the more formalized process.
>>
>> That's what I'm doing with my first review, anyway: just going through
>> the
>> TDC and evaluating the theme against each of the checklist points.
>>
>> I think, once we've got some context by reviewing an initial theme,
>> we'll
>> better be able to evaluate what criteria are critical/required
>> (GO/NO-GO),
>> and which ones are more subjective (1-5 rating, or whatever) - as well
>> as
>> how each subjective criterion (or category) should be weighted.
>>
>> Of course, my first theme assignment is taking a bit (sorry, Joseph -
>> the
>> one you gave me threw about five errors right off the bat!), so it may
>> not
>> be until the evening or tomorrow morning until I can report back on it.
>> In
>> the meantime, I'm appreciating the opportunity to read everyone else's
>> initial take on the review process.
>>
>> (As for getting the feedback to the theme authors themselves: I assume
>> Joseph is handling that?)
>>
>> Chip
>>
>> > Hi David,
>> >
>> > I agree with your points. I was thinking something like a scorecard
>> too.
>> >
>> > What I understood so far, is that we will go to check the theme
>> against
>> > the listed points from Theme_Development_Checklist, right? Is this our
>> > only job?
>> >
>> > I would appreciate a clarify from Joseph too.
>> >
>> > Regards,
>> > Daniel
>> >
>> >
>> >> Hi Joseph (and everyone else),
>> >>
>> >> First, is it Joseph or Joe?
>> >>
>> >> Second, can you clarify what exactly the goals are of a theme review?
>> >> What are we trying to accomplish? I realize we are trying to improve
>> the
>> >> quality and reliability of themes, but how is that going to work?
>> >> You've assigned some theme reviews to participants here, but they are
>> >> reporting "problems" back to us, not to the author to fix.  Is there
>> a
>> >> process I am not seeing?
>> >>
>> >> It seems to me, and I apologize if I am speaking out of turn, is that
>> >> all of the information on
>> >>
>> http://codex.wordpress.org/Theme_Development_Checklist#Theme_Unit_Test
>> >> could be turned into a scorecard.  It's already broken up into
>> sections.
>> >>  Give each bullet list a line item with a 1-5 rating or a y/n and set
>> a
>> >> comment section for free form comments at the end of each sub
>> section.
>> >> Then a theme submission could come in, be assigned to 3-5 of us, and
>> we
>> >> each complete a scorecard.  (heck portions of it could be automated.)
>> >> Then send these scorecards back to the theme author where they can
>> >> digest what they have done right, and what needs improvement.
>> Without
>> >> being too rigid, this process is crying for some formality of method.
>> >> This also gives the theme author feedback, but not at the whim of one
>> >> reviewer since some of the points (aesthetics in particular) are very
>> >> subjective.
>> >>
>> >> Just a thought...
>> >>
>> >> -David _______________________________________________
>> >> theme-reviewers mailing list
>> >> theme-reviewers at lists.wordpress.org
>> >> http://lists.wordpress.org/mailman/listinfo/theme-reviewers
>> >
>> > --
>> > Daniel Zilli _ Production Executive _ Jilliz Ltd.
>> >
>> > Phone: +66 (0) 85 334 1224 | Website: http://www.jilliz.com
>> > 288/43 BaanMai Puttabucha 36, Bangmod, Thungkru Bangkok 10140.
>> >
>> > _______________________________________________
>> > theme-reviewers mailing list
>> > theme-reviewers at lists.wordpress.org
>> > http://lists.wordpress.org/mailman/listinfo/theme-reviewers
>> >
>>
>>
>> _______________________________________________
>> theme-reviewers mailing list
>> theme-reviewers at lists.wordpress.org
>> http://lists.wordpress.org/mailman/listinfo/theme-reviewers
>> _______________________________________________
>> theme-reviewers mailing list
>> theme-reviewers at lists.wordpress.org
>> http://lists.wordpress.org/mailman/listinfo/theme-reviewers
>>
>>
>>
>>
>> _______________________________________________
>> theme-reviewers mailing list
>> theme-reviewers at lists.wordpress.org
>> http://lists.wordpress.org/mailman/listinfo/theme-reviewers
>>
>>
>>
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL:
>> <http://lists.wordpress.org/mailman/private/theme-reviewers/attachments/20100611/4542bba3/attachment.htm>
>>
>> ------------------------------
>>
>> _______________________________________________
>> theme-reviewers mailing list
>> theme-reviewers at lists.wordpress.org
>> http://lists.wordpress.org/mailman/listinfo/theme-reviewers
>>
>>
>> End of theme-reviewers Digest, Vol 1, Issue 32
>> **********************************************
>
>
> _______________________________________________
> theme-reviewers mailing list
> theme-reviewers at lists.wordpress.org
> http://lists.wordpress.org/mailman/listinfo/theme-reviewers
>
>
_______________________________________________
theme-reviewers mailing list
theme-reviewers at lists.wordpress.org
http://lists.wordpress.org/mailman/listinfo/theme-reviewers

More information about the theme-reviewers mailing list