[theme-reviewers] theme-reviewers Digest, Vol 1, Issue 32
Gavin Pearce
GavinP at tbs.uk.com
Fri Jun 11 15:33:32 UTC 2010
That said - wouldn't be too much of a job mind to update that
application so it was persistent, a little neater, and stored reviews
for each blog / user.
From: theme-reviewers-bounces at lists.wordpress.org
[mailto:theme-reviewers-bounces at lists.wordpress.org] On Behalf Of Tim
Golen
Sent: 11 June 2010 16:28
To: dronix
Cc: Edward Caissie; theme-reviewers
Subject: Re: [theme-reviewers] theme-reviewers Digest, Vol 1, Issue 32
The only problem I see with that is that it's not persistent. What I
mean is that where does the data go when you're done with it? That's why
I originally suggested the Google Docs Spreadsheets that would allow you
to collect the data just like the form that Edward gave an example of.
That would allow you to keep all the data and share it with the theme
author and other reviewers.
On Fri, Jun 11, 2010 at 9:16 AM, dronix <dronixs at gmail.com> wrote:
This looks good Edward, we could add additional text fields to each item
in the checklist for those who want to add any additional comments.
On Fri, Jun 11, 2010 at 11:12 AM, Edward Caissie
<edward.caissie at gmail.com> wrote:
I thought I had come across something of an interactive checklist a
while back, it just took me some time to find it in my bookmarks ...
have a look at this:
http://www.wplover.com/lab/theme-development-checklist
Might be a place to start from ...
Cais
On Fri, Jun 11, 2010 at 10:55 AM,
<theme-reviewers-request at lists.wordpress.org> wrote:
Send theme-reviewers mailing list submissions to
theme-reviewers at lists.wordpress.org
To subscribe or unsubscribe via the World Wide Web, visit
http://lists.wordpress.org/mailman/listinfo/theme-reviewers
or, via email, send a message with subject or body 'help' to
theme-reviewers-request at lists.wordpress.org
You can reach the person managing the list at
theme-reviewers-owner at lists.wordpress.org
When replying, please edit your Subject line so it is more specific
than "Re: Contents of theme-reviewers digest..."
Today's Topics:
1. Re: Goals and Process (Gavin Pearce)
----------------------------------------------------------------------
Message: 1
Date: Fri, 11 Jun 2010 15:43:01 +0100
From: "Gavin Pearce" <GavinP at tbs.uk.com>
Subject: Re: [theme-reviewers] Goals and Process
To: "theme-reviewers" <theme-reviewers at lists.wordpress.org>
Message-ID:
<CD47955B7E065D48AB5CF226C2ED75C0295BFE at tbs-sbs01.TBS.local>
Content-Type: text/plain; charset="utf-8"
Hi David,
Without trying to sound too awkward, I'm afraid I still disagree.
* The theme has a clear visual hierarchy. - It's either clear,
or it's not. If it's "almost" clear but not completely clear, then it's
"not clear" and would therefore be a no + advisory to improve slightly
if possible.
* The navigation is easy to understand. - It either easy to
understand, or it isn't. If it's "almost" easy to understand, but not
completely easy to understand, then it's a no and as above.
* Where possible, decorative images are in CSS. - All
decorative images are in the CSS .. if they are then yes, if not no +
another advisory.
* The theme uses CSS for all presentational aspects. - It either
uses CSS for all presentation aspects, or it doesn't.
Again, remember an advisory would be only for things that are minor, and
you'd have to get a few of them to fail.
As for things that are subjective on the eye of the reviewer, such as
your example - a "is that hierarchy 100% clear (yes/no)", is a lot
easier to resolve with a second pair of eyes than "is that hierarchy a 2
or a 3?".
I also feel, that while 'rating' things would be nice, we don't gain
anything extra by doing it.
E.g.
? The theme uses CSS for all presentational aspects.
No matter whether we rated this as a 2, 3 or 4, we'd still have the same
bit of advice to tell the author - "Put more of your presentational
images into the CSS".
Gav
//gavinpearce.com
From: david at mirmillo.com [mailto:david at mirmillo.com] On Behalf Of David
Damstra
Sent: 11 June 2010 15:25
To: Gavin Pearce
Cc: theme-reviewers
Subject: Re: [theme-reviewers] Goals and Process
Gavin, I completely agree about the Y/N flags... but there are several
items on the checklist that are truly subjective, for example:
* The theme has a clear visual hierarchy.
* The navigation is easy to understand.
* Where possible, decorative images are in CSS.
* The theme uses CSS for all presentational aspects.
Not to mention general aesthetic appeals and eye of the beholder stuff.
That is where I think your need rating scales. Ranging from not at all
(1) to For the most part(5) and address advisories in the free form
comments. I also think this is where it actually is good to have more
than one reviewer per theme, so that many eyes can see different issues,
and clear visual hierarchy can be interpreted different ways. This also
keeps creative license in the theme author's hands. Maybe they are
intending to create a dark navigation that I perceive as "not clear"
because my monitor is horribly calibrated.
Low subjective ratings without comments are worthless, use the comments
to explain the rating and recommend a course of action and constructive
criticism. But the subjective portions are advice for improvement.
Whereas the Y/N portions of the site are more make it or break it. If
you don't pass the critical Y/N portions (or at least a minimum of them)
then the theme is no fly until they are fixed.
Again, I am just talking here... I am still unclear on what the expected
outcome of these theme reviews are. Hoping to hear some clarification
from Joseph.
On Fri, Jun 11, 2010 at 10:11 AM, Gavin Pearce <GavinP at tbs.uk.com>
wrote:
Thanks for the feedback Tim, some great ideas with the how to format it.
I also agree with Devon earlier, though a PDF would be handy, somewhere
that responses can also be displayed to the original author would work
quicker. A nice online app would be handy indeed.
With the rating system though - I feel that also the subjective items
should be a case of just simply pass/fail - largely because by their
very nature, subjective items are somewhat subjective (excuse the pun).
Is the theme widgetized as fully as possible?
Are comments displayed correctly?
On a scale of 1 - 5 would be very hard to judge consistently between
reviewers. What makes a widgetized theme a 4 rather than a 3 for
example? Also saying that, from past experience you'll also just end up
with a lot of 3's.
If a template has a MAJOR problem, it fails the review instantly.
If a template has a MINOR problem, we issue an advisory to the author to
make some changes, but continue to allow it.
If a template has more than X advisories, it also fails.
That way, a template that isn't fully widgetized will pass if that is
it's only problem, but a template that isn't fully widgetized, has
errors in the comment area, has formatting errors + others will fail.
Simple yes/no checklist is the way forward think, especially in terms
of simplicity, consistency and quality, for both the theme authors, and
ourseleves.
Feel free to tell me I'm completely wrong though. ; )
Gav
/gavinpearce.com
From: tgolen at gmail.com [mailto:tgolen at gmail.com] On Behalf Of Tim Golen
Sent: 11 June 2010 14:44
To: Gavin Pearce
Cc: theme-reviewers
Subject: Re: [theme-reviewers] Goals and Process
Hey Gav,
I think what you say makes perfect sense, and I've liked where the
discussion is headed. I too wondered about the validity of 5 star rating
system for each item and for a lot of items in the checklist it's either
black or white... either it works or it doesn't work. However, I think
there are also items that could use a 5-point rating system because they
are a bit more subjective. Such as "Are comments displayed correctly"
and "Is the theme widgetized as fully as possible?".
I think the organization of the review process could be done to extreme
(using a bug tracking system like Bugzilla which would be a very
effective tool for communication with reviewers and authors), but if you
wanted a really simple solution you could also use a Google Docs
Spreadsheet with the Form functionality. It would only take a short
amount of a time to setup a form for the entire checklist. You could use
checkboxes for the black-and-white items, and you could use a 5 point
radio group for the more subjective items. Along with a text field for
the name of the theme and free form comments at the bottom of each
sub-section. With this solution you could use one spreadsheet per theme,
or have all theme reviews in a single spreadsheet (they have some cool
summarization views for the form data). Just a thought... the
communication channels would still remain unresolved at that point.
Tim
On Fri, Jun 11, 2010 at 7:07 AM, Gavin Pearce <GavinP at tbs.uk.com> wrote:
Agreed almost completely.
Not convinced about 1 - 5 ratings mind, you start to open up the process
to the whim of individual reviewers again.
Checklist should check for items that are:
* Critical items - No go, instant fail..
* Less critical - You get a 'advisory'.
If you're template has more than X amount of "advisories", it doesn't
pass. (Similar to the format of the UK Driving Test).
I guess you could also get a different amount of "advisory points"
depending on the problem, but it starts getting overly complicated
again. Personally I think keep it as simple as possible - this will
result in a much better consistency of what quality of template we
allow/disallow.
If built around this model, you can start to add extra criteria to
improve the general quality of all templates, e.g.:
- To get a V2 version of the same template into the system, the
author first has to demonstrate they acted upon at least some of the
advisories issued first time round.
Thoughts? :)
Gav
/gavinpearce.com
-----Original Message-----
From: theme-reviewers-bounces at lists.wordpress.org
[mailto:theme-reviewers-bounces at lists.wordpress.org] On Behalf Of
chip at chipbennett.net
Sent: 11 June 2010 13:43
To: theme-reviewers at lists.wordpress.org
Subject: Re: [theme-reviewers] Goals and Process
I concur.
I'm thinking that Joseph (Joe?) is getting several people started on an
initial theme review, in order to give us some better context for coming
up with the more formalized process.
That's what I'm doing with my first review, anyway: just going through
the
TDC and evaluating the theme against each of the checklist points.
I think, once we've got some context by reviewing an initial theme,
we'll
better be able to evaluate what criteria are critical/required
(GO/NO-GO),
and which ones are more subjective (1-5 rating, or whatever) - as well
as
how each subjective criterion (or category) should be weighted.
Of course, my first theme assignment is taking a bit (sorry, Joseph -
the
one you gave me threw about five errors right off the bat!), so it may
not
be until the evening or tomorrow morning until I can report back on it.
In
the meantime, I'm appreciating the opportunity to read everyone else's
initial take on the review process.
(As for getting the feedback to the theme authors themselves: I assume
Joseph is handling that?)
Chip
> Hi David,
>
> I agree with your points. I was thinking something like a scorecard
too.
>
> What I understood so far, is that we will go to check the theme
against
> the listed points from Theme_Development_Checklist, right? Is this our
> only job?
>
> I would appreciate a clarify from Joseph too.
>
> Regards,
> Daniel
>
>
>> Hi Joseph (and everyone else),
>>
>> First, is it Joseph or Joe?
>>
>> Second, can you clarify what exactly the goals are of a theme review?
>> What are we trying to accomplish? I realize we are trying to improve
the
>> quality and reliability of themes, but how is that going to work?
>> You've assigned some theme reviews to participants here, but they are
>> reporting "problems" back to us, not to the author to fix. Is there
a
>> process I am not seeing?
>>
>> It seems to me, and I apologize if I am speaking out of turn, is that
>> all of the information on
>>
http://codex.wordpress.org/Theme_Development_Checklist#Theme_Unit_Test
>> could be turned into a scorecard. It's already broken up into
sections.
>> Give each bullet list a line item with a 1-5 rating or a y/n and set
a
>> comment section for free form comments at the end of each sub
section.
>> Then a theme submission could come in, be assigned to 3-5 of us, and
we
>> each complete a scorecard. (heck portions of it could be automated.)
>> Then send these scorecards back to the theme author where they can
>> digest what they have done right, and what needs improvement.
Without
>> being too rigid, this process is crying for some formality of method.
>> This also gives the theme author feedback, but not at the whim of one
>> reviewer since some of the points (aesthetics in particular) are very
>> subjective.
>>
>> Just a thought...
>>
>> -David _______________________________________________
>> theme-reviewers mailing list
>> theme-reviewers at lists.wordpress.org
>> http://lists.wordpress.org/mailman/listinfo/theme-reviewers
>
> --
> Daniel Zilli _ Production Executive _ Jilliz Ltd.
>
> Phone: +66 (0) 85 334 1224 | Website: http://www.jilliz.com
> 288/43 BaanMai Puttabucha 36, Bangmod, Thungkru Bangkok 10140.
>
> _______________________________________________
> theme-reviewers mailing list
> theme-reviewers at lists.wordpress.org
> http://lists.wordpress.org/mailman/listinfo/theme-reviewers
>
_______________________________________________
theme-reviewers mailing list
theme-reviewers at lists.wordpress.org
http://lists.wordpress.org/mailman/listinfo/theme-reviewers
_______________________________________________
theme-reviewers mailing list
theme-reviewers at lists.wordpress.org
http://lists.wordpress.org/mailman/listinfo/theme-reviewers
_______________________________________________
theme-reviewers mailing list
theme-reviewers at lists.wordpress.org
http://lists.wordpress.org/mailman/listinfo/theme-reviewers
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.wordpress.org/mailman/private/theme-reviewers/attachments/
20100611/4542bba3/attachment.htm>
------------------------------
_______________________________________________
theme-reviewers mailing list
theme-reviewers at lists.wordpress.org
http://lists.wordpress.org/mailman/listinfo/theme-reviewers
End of theme-reviewers Digest, Vol 1, Issue 32
**********************************************
_______________________________________________
theme-reviewers mailing list
theme-reviewers at lists.wordpress.org
http://lists.wordpress.org/mailman/listinfo/theme-reviewers
_______________________________________________
theme-reviewers mailing list
theme-reviewers at lists.wordpress.org
http://lists.wordpress.org/mailman/listinfo/theme-reviewers
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wordpress.org/mailman/private/theme-reviewers/attachments/20100611/fd6f4423/attachment-0001.htm>
More information about the theme-reviewers
mailing list