[theme-reviewers] theme-reviewers Digest, Vol 1, Issue 32

dronix dronixs at gmail.com
Fri Jun 11 15:33:49 UTC 2010 

We can set up a database for this interactive checkist, this way we could
have the results publicly available or just to the theme author.

On Fri, Jun 11, 2010 at 11:27 AM, Tim Golen <tim at golen.net> wrote:

> The only problem I see with that is that it's not persistent. What I mean
> is that where does the data go when you're done with it? That's why I
> originally suggested the Google Docs Spreadsheets that would allow you to
> collect the data just like the form that Edward gave an example of. That
> would allow you to keep all the data and share it with the theme author and
> other reviewers.
>
>
> On Fri, Jun 11, 2010 at 9:16 AM, dronix <dronixs at gmail.com> wrote:
>
>> This looks good Edward, we could add additional text fields to each item
>> in the checklist for those who want to add any additional comments.
>>
>>
>> On Fri, Jun 11, 2010 at 11:12 AM, Edward Caissie <
>> edward.caissie at gmail.com> wrote:
>>
>>> I thought I had come across something of an interactive checklist a while
>>> back, it just took me some time to find it in my bookmarks ... have a look
>>> at this: http://www.wplover.com/lab/theme-development-checklist
>>>
>>> Might be a place to start from ...
>>>
>>>
>>> Cais
>>>
>>> On Fri, Jun 11, 2010 at 10:55 AM, <
>>> theme-reviewers-request at lists.wordpress.org> wrote:
>>>
>>>> Send theme-reviewers mailing list submissions to
>>>>        theme-reviewers at lists.wordpress.org
>>>>
>>>> To subscribe or unsubscribe via the World Wide Web, visit
>>>>        http://lists.wordpress.org/mailman/listinfo/theme-reviewers
>>>> or, via email, send a message with subject or body 'help' to
>>>>        theme-reviewers-request at lists.wordpress.org
>>>>
>>>> You can reach the person managing the list at
>>>>        theme-reviewers-owner at lists.wordpress.org
>>>>
>>>> When replying, please edit your Subject line so it is more specific
>>>> than "Re: Contents of theme-reviewers digest..."
>>>>
>>>>
>>>> Today's Topics:
>>>>
>>>>   1. Re: Goals and Process (Gavin Pearce)
>>>>
>>>>
>>>> ----------------------------------------------------------------------
>>>>
>>>> Message: 1
>>>> Date: Fri, 11 Jun 2010 15:43:01 +0100
>>>> From: "Gavin Pearce" <GavinP at tbs.uk.com>
>>>> Subject: Re: [theme-reviewers] Goals and Process
>>>> To: "theme-reviewers" <theme-reviewers at lists.wordpress.org>
>>>> Message-ID:
>>>>        <CD47955B7E065D48AB5CF226C2ED75C0295BFE at tbs-sbs01.TBS.local>
>>>> Content-Type: text/plain; charset="utf-8"
>>>>
>>>> Hi David,
>>>>
>>>>
>>>>
>>>> Without trying to sound too awkward, I'm afraid I still disagree.
>>>>
>>>> *       The theme has a clear visual hierarchy.   - It's either clear,
>>>> or it's not. If it's "almost" clear but not completely clear, then it's "not
>>>> clear" and would therefore be a no + advisory to improve slightly if
>>>> possible.
>>>> *       The navigation is easy to understand.   - It either easy to
>>>> understand, or it isn't. If it's "almost" easy to understand, but not
>>>> completely easy to understand, then it's a no and as above.
>>>>
>>>> *       Where possible, decorative images are in CSS.    - All
>>>> decorative images are in the CSS .. if they are then yes, if not no +
>>>> another advisory.
>>>> *       The theme uses CSS for all presentational aspects.  - It either
>>>> uses CSS for all presentation aspects, or it doesn't.
>>>>
>>>> Again, remember an advisory would be only for things that are minor, and
>>>> you'd have to get a few of them to fail.
>>>>
>>>> As for things that are subjective on the eye of the reviewer, such as
>>>> your example - a "is that hierarchy 100% clear (yes/no)", is a lot easier to
>>>> resolve with a second pair of eyes than "is that hierarchy a 2 or a 3?".
>>>>
>>>> I also feel, that while 'rating' things would be nice, we don't gain
>>>> anything extra by doing it.
>>>>
>>>> E.g.
>>>>
>>>> ?         The theme uses CSS for all presentational aspects.
>>>>
>>>> No matter whether we rated this as a 2, 3 or 4, we'd still have the same
>>>> bit of advice to tell the author - "Put more of your presentational images
>>>> into the CSS".
>>>>
>>>> Gav
>>>> //gavinpearce.com
>>>>
>>>>
>>>>
>>>> From: david at mirmillo.com [mailto:david at mirmillo.com] On Behalf Of David
>>>> Damstra
>>>> Sent: 11 June 2010 15:25
>>>> To: Gavin Pearce
>>>> Cc: theme-reviewers
>>>> Subject: Re: [theme-reviewers] Goals and Process
>>>>
>>>>
>>>>
>>>> Gavin, I completely agree about the Y/N flags... but there are several
>>>> items on the checklist that are truly subjective, for example:
>>>>
>>>> *       The theme has a clear visual hierarchy.
>>>> *       The navigation is easy to understand.
>>>>
>>>> *       Where possible, decorative images are in CSS.
>>>> *       The theme uses CSS for all presentational aspects.
>>>>
>>>> Not to mention general aesthetic appeals and eye of the beholder stuff.
>>>>  That is where I think your need rating scales.  Ranging from not at all (1)
>>>> to For the most part(5) and address advisories in the free form comments.  I
>>>> also think this is where it actually is good to have more than one reviewer
>>>> per theme, so that many eyes can see different issues, and clear visual
>>>> hierarchy can be interpreted different ways.  This also keeps creative
>>>> license in the theme author's hands.  Maybe they are intending to create a
>>>> dark navigation that I perceive as "not clear" because my monitor is
>>>> horribly calibrated.
>>>>
>>>> Low subjective ratings without comments are worthless, use the comments
>>>> to explain the rating and recommend a course of action and constructive
>>>> criticism.  But the subjective portions are advice for improvement.  Whereas
>>>> the Y/N portions of the site are more make it or break it.  If you don't
>>>> pass the critical Y/N portions (or at least a minimum of them) then the
>>>> theme is no fly until they are fixed.
>>>>
>>>> Again, I am just talking here... I am still unclear on what the expected
>>>> outcome of these theme reviews are. Hoping to hear some clarification from
>>>> Joseph.
>>>>
>>>>
>>>>
>>>>
>>>> On Fri, Jun 11, 2010 at 10:11 AM, Gavin Pearce <GavinP at tbs.uk.com>
>>>> wrote:
>>>>
>>>> Thanks for the feedback Tim, some great ideas with the how to format it.
>>>> I also agree with Devon earlier, though a PDF would be handy, somewhere that
>>>> responses can also be displayed to the original author would work quicker.
>>>>  A nice online app would be handy indeed.
>>>>
>>>>
>>>>
>>>> With the rating system though - I feel that also the subjective items
>>>> should be a case of just simply pass/fail - largely because by their very
>>>> nature, subjective items are somewhat subjective (excuse the pun).
>>>>
>>>>
>>>>
>>>> Is the theme widgetized as fully as possible?
>>>>
>>>> Are comments displayed correctly?
>>>>
>>>>
>>>>
>>>> On a scale of 1 - 5 would be very hard to judge consistently between
>>>> reviewers. What makes a widgetized theme a 4 rather than a 3 for example?
>>>> Also saying that, from past experience you'll also just end up with a lot of
>>>> 3's.
>>>>
>>>>
>>>>
>>>> If a template has a MAJOR problem, it fails the review instantly.
>>>>
>>>> If a template has a MINOR problem, we issue an advisory to the author to
>>>> make some changes, but continue to allow it.
>>>>
>>>> If a template has more than X advisories, it also fails.
>>>>
>>>>
>>>>
>>>> That way, a template that isn't fully widgetized will pass if that is
>>>> it's only problem, but a template that isn't fully widgetized, has errors in
>>>> the comment area, has formatting errors + others will fail.
>>>>
>>>>
>>>>
>>>> Simple yes/no checklist is the way forward  think, especially in terms
>>>> of simplicity, consistency and quality, for both the theme authors, and
>>>> ourseleves.
>>>>
>>>>
>>>>
>>>> Feel free to tell me I'm completely wrong though.  ;  )
>>>>
>>>>
>>>>
>>>> Gav
>>>>
>>>> /gavinpearce.com
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> From: tgolen at gmail.com [mailto:tgolen at gmail.com] On Behalf Of Tim Golen
>>>> Sent: 11 June 2010 14:44
>>>> To: Gavin Pearce
>>>> Cc: theme-reviewers
>>>>
>>>>
>>>> Subject: Re: [theme-reviewers] Goals and Process
>>>>
>>>>
>>>>
>>>> Hey Gav,
>>>>
>>>>
>>>>
>>>> I think what you say makes perfect sense, and I've liked where the
>>>> discussion is headed. I too wondered about the validity of 5 star rating
>>>> system for each item and for a lot of items in the checklist it's either
>>>> black or white... either it works or it doesn't work. However, I think there
>>>> are also items that could use a 5-point rating system because they are a bit
>>>> more subjective. Such as "Are comments displayed correctly" and "Is the
>>>> theme widgetized as fully as possible?".
>>>>
>>>>
>>>>
>>>> I think the organization of the review process could be done to extreme
>>>> (using a bug tracking system like Bugzilla which would be a very effective
>>>> tool for communication with reviewers and authors), but if you wanted a
>>>> really simple solution you could also use a Google Docs Spreadsheet with the
>>>> Form functionality. It would only take a short amount of a time to setup a
>>>> form for the entire checklist. You could use checkboxes for the
>>>> black-and-white items, and you could use a 5 point radio group for the more
>>>> subjective items. Along with a text field for the name of the theme and free
>>>> form comments at the bottom of each sub-section. With this solution you
>>>> could use one spreadsheet per theme, or have all theme reviews in a single
>>>> spreadsheet (they have some cool summarization views for the form data).
>>>> Just a thought... the communication channels would still remain unresolved
>>>> at that point.
>>>>
>>>>
>>>>
>>>> Tim
>>>>
>>>> On Fri, Jun 11, 2010 at 7:07 AM, Gavin Pearce <GavinP at tbs.uk.com>
>>>> wrote:
>>>>
>>>> Agreed almost completely.
>>>>
>>>> Not convinced about 1 - 5 ratings mind, you start to open up the process
>>>> to the whim of individual reviewers again.
>>>>
>>>> Checklist should check for items that are:
>>>>
>>>>       * Critical items - No go, instant fail..
>>>>       * Less critical - You get a 'advisory'.
>>>>
>>>> If you're template has more than X amount of "advisories", it doesn't
>>>> pass. (Similar to the format of the UK Driving Test).
>>>>
>>>> I guess you could also get a different amount of "advisory points"
>>>> depending on the problem, but it starts getting overly complicated
>>>> again. Personally I think keep it as simple as possible - this will
>>>> result in a much better consistency of what quality of template we
>>>> allow/disallow.
>>>>
>>>> If built around this model, you can start to add extra criteria to
>>>> improve the general quality of all templates, e.g.:
>>>>       - To get a V2 version of the same template into the system, the
>>>> author first has to demonstrate they acted upon at least some of the
>>>> advisories issued first time round.
>>>>
>>>>
>>>> Thoughts?       :)
>>>>
>>>>
>>>> Gav
>>>> /gavinpearce.com
>>>>
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: theme-reviewers-bounces at lists.wordpress.org
>>>> [mailto:theme-reviewers-bounces at lists.wordpress.org] On Behalf Of
>>>> chip at chipbennett.net
>>>> Sent: 11 June 2010 13:43
>>>> To: theme-reviewers at lists.wordpress.org
>>>> Subject: Re: [theme-reviewers] Goals and Process
>>>>
>>>> I concur.
>>>>
>>>> I'm thinking that Joseph (Joe?) is getting several people started on an
>>>> initial theme review, in order to give us some better context for coming
>>>> up with the more formalized process.
>>>>
>>>> That's what I'm doing with my first review, anyway: just going through
>>>> the
>>>> TDC and evaluating the theme against each of the checklist points.
>>>>
>>>> I think, once we've got some context by reviewing an initial theme,
>>>> we'll
>>>> better be able to evaluate what criteria are critical/required
>>>> (GO/NO-GO),
>>>> and which ones are more subjective (1-5 rating, or whatever) - as well
>>>> as
>>>> how each subjective criterion (or category) should be weighted.
>>>>
>>>> Of course, my first theme assignment is taking a bit (sorry, Joseph -
>>>> the
>>>> one you gave me threw about five errors right off the bat!), so it may
>>>> not
>>>> be until the evening or tomorrow morning until I can report back on it.
>>>> In
>>>> the meantime, I'm appreciating the opportunity to read everyone else's
>>>> initial take on the review process.
>>>>
>>>> (As for getting the feedback to the theme authors themselves: I assume
>>>> Joseph is handling that?)
>>>>
>>>> Chip
>>>>
>>>> > Hi David,
>>>> >
>>>> > I agree with your points. I was thinking something like a scorecard
>>>> too.
>>>> >
>>>> > What I understood so far, is that we will go to check the theme
>>>> against
>>>> > the listed points from Theme_Development_Checklist, right? Is this our
>>>> > only job?
>>>> >
>>>> > I would appreciate a clarify from Joseph too.
>>>> >
>>>> > Regards,
>>>> > Daniel
>>>> >
>>>> >
>>>> >> Hi Joseph (and everyone else),
>>>> >>
>>>> >> First, is it Joseph or Joe?
>>>> >>
>>>> >> Second, can you clarify what exactly the goals are of a theme review?
>>>> >> What are we trying to accomplish? I realize we are trying to improve
>>>> the
>>>> >> quality and reliability of themes, but how is that going to work?
>>>> >> You've assigned some theme reviews to participants here, but they are
>>>> >> reporting "problems" back to us, not to the author to fix.  Is there
>>>> a
>>>> >> process I am not seeing?
>>>> >>
>>>> >> It seems to me, and I apologize if I am speaking out of turn, is that
>>>> >> all of the information on
>>>> >>
>>>> http://codex.wordpress.org/Theme_Development_Checklist#Theme_Unit_Test
>>>> >> could be turned into a scorecard.  It's already broken up into
>>>> sections.
>>>> >>  Give each bullet list a line item with a 1-5 rating or a y/n and set
>>>> a
>>>> >> comment section for free form comments at the end of each sub
>>>> section.
>>>> >> Then a theme submission could come in, be assigned to 3-5 of us, and
>>>> we
>>>> >> each complete a scorecard.  (heck portions of it could be automated.)
>>>> >> Then send these scorecards back to the theme author where they can
>>>> >> digest what they have done right, and what needs improvement.
>>>> Without
>>>> >> being too rigid, this process is crying for some formality of method.
>>>> >> This also gives the theme author feedback, but not at the whim of one
>>>> >> reviewer since some of the points (aesthetics in particular) are very
>>>> >> subjective.
>>>> >>
>>>> >> Just a thought...
>>>> >>
>>>> >> -David _______________________________________________
>>>> >> theme-reviewers mailing list
>>>> >> theme-reviewers at lists.wordpress.org
>>>> >> http://lists.wordpress.org/mailman/listinfo/theme-reviewers
>>>> >
>>>> > --
>>>> > Daniel Zilli _ Production Executive _ Jilliz Ltd.
>>>> >
>>>> > Phone: +66 (0) 85 334 1224 | Website: http://www.jilliz.com
>>>> > 288/43 BaanMai Puttabucha 36, Bangmod, Thungkru Bangkok 10140.
>>>> >
>>>> > _______________________________________________
>>>> > theme-reviewers mailing list
>>>> > theme-reviewers at lists.wordpress.org
>>>> > http://lists.wordpress.org/mailman/listinfo/theme-reviewers
>>>> >
>>>>
>>>>
>>>> _______________________________________________
>>>> theme-reviewers mailing list
>>>> theme-reviewers at lists.wordpress.org
>>>> http://lists.wordpress.org/mailman/listinfo/theme-reviewers
>>>> _______________________________________________
>>>> theme-reviewers mailing list
>>>> theme-reviewers at lists.wordpress.org
>>>> http://lists.wordpress.org/mailman/listinfo/theme-reviewers
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> theme-reviewers mailing list
>>>> theme-reviewers at lists.wordpress.org
>>>> http://lists.wordpress.org/mailman/listinfo/theme-reviewers
>>>>
>>>>
>>>>
>>>> -------------- next part --------------
>>>> An HTML attachment was scrubbed...
>>>> URL: <
>>>> http://lists.wordpress.org/mailman/private/theme-reviewers/attachments/20100611/4542bba3/attachment.htm
>>>> >
>>>>
>>>> ------------------------------
>>>>
>>>> _______________________________________________
>>>> theme-reviewers mailing list
>>>> theme-reviewers at lists.wordpress.org
>>>> http://lists.wordpress.org/mailman/listinfo/theme-reviewers
>>>>
>>>>
>>>> End of theme-reviewers Digest, Vol 1, Issue 32
>>>> **********************************************
>>>>
>>>
>>>
>>> _______________________________________________
>>> theme-reviewers mailing list
>>> theme-reviewers at lists.wordpress.org
>>> http://lists.wordpress.org/mailman/listinfo/theme-reviewers
>>>
>>>
>>
>> _______________________________________________
>> theme-reviewers mailing list
>> theme-reviewers at lists.wordpress.org
>> http://lists.wordpress.org/mailman/listinfo/theme-reviewers
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wordpress.org/mailman/private/theme-reviewers/attachments/20100611/7fdf245d/attachment-0001.htm>

More information about the theme-reviewers mailing list