[wp-hackers] WordPress plugin inspections

Wed Feb 19 21:32:52 UTC 2014

Hi Chris,

I think you raise some good points.

I'm going to change the commissioning bit. I agree that it is prone to 
misinterpretation.

I think there's no way to do this without some caveating, and I think it 
is useful notwithstanding the caveats. But I agree that they're perhaps 
not clear enough, and that "unsafe for use" might be a little too 
categorical. I've had a couple of emails privately from developers this 
evening which do make it pretty plain that the current approach is 
confusing people.

We have published a clear and transparent set of criteria, and those are 
what we apply. We have done our best to make this as clear as possible.

We are doing our best to do something useful, and I strongly believe 
that this site goes some way to filling an important gap. On this we may 
perhaps have to agree to disagree. But I do value the points you've made 
and we will make some changes based upon then. I'd be keen to hear any 
other feedback you might have later (short of "stop doing it"!)

Harry

On 19/02/2014 21:25, Chris Williams wrote:
> You are filled with double-speak.  You say you don't want to make
> categorical claims about people's code, but that's exactly what you do.
> You say you don't want to hold people ransom, but then below admit that's
> precisely what you are trying to do to make a business out of it.  You say
> your inspections aren't worth the bits it takes to express them, but then
> highlight the results in red, with vague caveats to try to hedge your bets.
>
> Here's my advice:  Keep your opinions to yourself (and maybe your clients).
>
> Unless and until you provide a clear/clean/transparent method for
> evaluation, that isn't subject to the many issues I raised in my other
> note, that has well-defined and very public methods for redress, and that
> doesn't result in random and spurious claims about code you've barely even
> bothered to understand, keep mum.  As your grandmother said "if you can't
> say anything nice, don't say anything at all."
>
> Want to see how ugly this can get?  Enable comments on your reviews, and
> sit back and watch.  You'll get plugin developers passionately trying to
> defend themselves.  You'll get their competitors jumping in to throw mud.
> You'll get fanboys and haters jumping all over each other.  It will get so
> ugly so fast...
>
> I appreciate that you feel there is some value to you and your clients in
> this work.  But please, don't try to be the "Good Housekeeping" seal of
> approval without a whole lot more thought into it than you've clearly done
> so far...
>
> On 2/19/14 12:52 PM, "Harry Metcalfe" <harry at dxw.com> wrote:
>
>> Hi Chris,
>>
>> I'm sorry you feel that way, and I can say categorically that we are not
>> trying to hold anyone to ransom. I'll try to explain.
>>
>> Going back a couple of years, our clients expected us to give them some
>> sort of assessment of plugins before we suggested using them. For a
>> while, we did this informally, and the results were very mixed. We found
>> that sometimes we would miss things. There was no set of criteria that
>> we applied, and we didn't record the results. This also led us to waste
>> time by checking out the same problem twice.
>>
>> To solve these problems, we decided have a list of things that we think
>> are important (https://security.dxw.com/about/plugin-inspections/) and
>> to record the results of inspections somewhere so we didn't duplicate
>> work. We did this in private for a while but then thought that this was
>> probably information that others might find useful. So, we decided to
>> publish the results.
>>
>> We have tried very hard to make sure that the results of these
>> inspections, and our confidence in them, is obvious to people who read
>> them. We've published the process. We've made sure it's clear that
>> inspections deal with likelihoods, not certainties. We've said that
>> people should always conduct their own checks. We've set out our terms
>> of service prominently, which include contact information for anyone
>> who'd like to tell us we're wrong. And we're totally happy (within
>> reason) to revisit things if people do that.
>>
>> I would very much like it if these inspections could be more thorough,
>> but unfortunately, we're subject to the same commercial realities as
>> everyone else. We care more about security than most of our clients.
>> Most people are not willing to pay for security assurance work.
>> Inspections are light-touch because we don't charge existing clients for
>> them, and that's the only way we can make it economical.
>>
>> I hope we can figure out some way to make some money out of this (hence
>> those messages saying we can be commissioned) but so far, we haven't
>> made a penny. We're just trying to make the outputs of something we do
>> anyway useful to a wider group.
>>
>> If you have feedback on practical ways we could do that better, I'd love
>> to hear it.
>>
>> Harry
>>
>> On 19/02/2014 20:17, Chris Williams wrote:
>>> I certainly can't speak for others, but I would venture to say that your
>>> business model is evil at best.  You do fly-by character assassination
>>> (oops, I mean "light-touch inspections"), based on personal bias ("this
>>> plugin is large"), and then broadly publish the results as if they are
>>> somehow authoritative.  Worse yet, you then hold plugin developers at
>>> ransom for changing the review: "If you would like to commission us to
>>> inspect or review the latest version, please contact us."
>>>
>>> How this is of value to anyone, and how you sleep at night with this
>>> specious business model, is completely beyond me.
>>>
>>> On 2/19/14 10:43 AM, "Harry Metcalfe" <harry at dxw.com> wrote:
>>>
>>>> Hello list,
>>>>
>>>> We write and publish light-touch inspections of WordPress plugins that
>>>> we do for our clients. They are just a guide - we conduct some basic
>>>> checks, not a thorough review.
>>>>
>>>> Would plugins which fail this inspection be of general interest to the
>>>> list and therefore worth posting? Is the list also interested in
>>>> vulnerability advisories, or do people tend to get those elsewhere?
>>>>
>>>> Here's an example report:
>>>>
>>>> https://security.dxw.com/plugins/pods-custom-content-types-and-fields/
>>>>
>>>> Grateful for a steer...
>>>>
>>>> Harry
>>>>
>>>>
>>>> -- 
>>>> Harry Metcalfe
>>>> 07790 559 876
>>>> @harrym
>>>>
>>>> _______________________________________________
>>>> wp-hackers mailing list
>>>> wp-hackers at lists.automattic.com
>>>> http://lists.automattic.com/mailman/listinfo/wp-hackers
>>> _______________________________________________
>>> wp-hackers mailing list
>>> wp-hackers at lists.automattic.com
>>> http://lists.automattic.com/mailman/listinfo/wp-hackers
>> _______________________________________________
>> wp-hackers mailing list
>> wp-hackers at lists.automattic.com
>> http://lists.automattic.com/mailman/listinfo/wp-hackers
> _______________________________________________
> wp-hackers mailing list
> wp-hackers at lists.automattic.com
> http://lists.automattic.com/mailman/listinfo/wp-hackers