[wp-meta] [Making WordPress.org] #8009: Phased releases and roll-outs of plugins

[Making WordPress.org]

#8009: Phased releases and roll-outs of plugins
------------------------------+---------------------
 Reporter:  matt              |       Owner:  (none)
     Type:  enhancement       |      Status:  new
 Priority:  normal            |   Milestone:
Component:  Plugin Directory  |  Resolution:
 Keywords:  has-patch         |
------------------------------+---------------------

Comment (by azaozz):

 Replying to [comment:11 dd32]:

 Great way to put this together, thanks! I'd like to add few
 ideas/suggestions, hope they make sense.

 > The most obvious way to segment the sites is to extract the sites
 domain...
 > There's a shortcoming here though, a site will always be in the same
 bucket of sites

 Thinking that a good way to implement this (at least at first) would be to
 keep it as simple as possible. Imho the main consideration here is that
 choosing which sites will get update notifications early and which will
 get delayed is not critical. Preferably it would be random, but no need to
 enforce only one try per site per day, etc.

 In that terms think there is no need of buckets or storing any site
 identifiers. Only thing that needs to be stored is how many sites were
 actually updated. Seems this data would be coming from the downloads
 counter. In my mind perhaps the simplest way this could work is:

 1. When a plugin update gets released and sites check for updates the
 wp.org API would offer an update as usual. Can also be throttled but not
 sure if it will be needed. Depends on how often the download counter is
 updated/checked, see below.

 2. At update release time a number of initial updates is (pre)determined
 and stored in memory. This could be percentage based, or better a
 predetermined number. Download count of the new version is checked against
 this number every minute of so. Once the number is reached (or exceeded)
 the API would stop to offer the update.

 3. If there is a problem with the release the plugin authors would be able
 to pause or suspend it. If no errors the API would automatically continue
 to offer the update (perhaps) 24 hours after the initial release.

 I see many people here suggest more control and features for plugin
 authors, but perhaps starting simple would be best. Thinking we can come
 up with good initial defaults that will work satisfactory for most
 plugins. For example I agree that phased releases would not make sense for
 plugins with less than 10k active installs, perhaps even 20k. Another good
 default would probably be the auto-continue time set at 24 hours.

 > As mentioned in several comments, for this to be effective we need some
 way of surfacing errors back to the authors.

 I've been maintaining quite a few plugins through the years and have
 always depended on the plugins support forums. It's true that the forums
 for plugins with many active installs are way more lively, but even 1-2
 comments soon after a new release should be enough to draw the author's
 attention. Imho this can be considered an adequate way of notification for
 now.

 > I agree that it should be for auto-updates only.

 The simplest way to do this would be to stop offering the update as soon
 as the number of downloads reaches the number we initially intended to
 update. Then the API would auto-continue to offer the update 24 hours
 later (TBD).

 I agree it would be nice to have "manual override" where the site admins
 will be able to update a plugin despite that the number of initial updates
 has been reached and the API doesn't offer updates at that time. Of course
 they will need to be informed about what they are doing. This would have
 to be implemented in WP. Any takers? :)

 > Instead of hashing the domain, we could hash the entire user-agent
 header. That would result in the bucket changing each time

 Don't really think it matters much if the API skips offering the update on
 one check and offers it at the next. Not critical by any means.

 > Using a count threshold **could** be better. However, we'd have to store
 the number of sites we've offered each update

 The (simplest?) way this may work is to have a pre-determined number of
 "early updates" for each plugin release, then check it against the
 download counter for the updated version (see above). That way it doesn't
 matter which sites we offered the update to. If they were updated we're
 all done and the site admins won't see the update notifications any more.

 > My original intention here was to delay implementation of this into
 WordPress 6.9, however, we still need to support WordPress 2.8 ~ 6.8
 clients anyway so we have to take those into account.

 Yea, backporting this may be difficult (if we want to allow site admins to
 manually override). If not, thinking the code would be the same for new
 and old versions of WP. I.e. it will be controlled by the API.

 > The major shortcoming of the existing WordPress plugin update check is
 that we currently cannot provide different details for auto-updates vs
 user-initiated updates.

 Right. Thinking it would be helpful for the API to be able to tell WP (the
 client) to auto-update or not to auto-update. But seems this can be
 initially implemented without that functionality.

 > The WordPress.org plugin update API can return one of 4 states:
 > 1. Plugin XYZ is not hosted on WordPress.org
 > 2. Plugin XYZ is hosted on WordPress.org, and you're running the latest
 version: 1.0
 > 3. Plugin XYZ is hosted on WordPress.org, and you're running an outdated
 version: 1.0. Version 2.0 is the latest.
 > 4. Plugin XYZ is hosted on WordPress.org, you're running an outdated
 version (1.0 vs 2.0) BUT automatic updates (both user-initiated and
 background auto-updates) are not available for this plugin.
 >
 > By returning 4, WordPress will display a notice such as the following:
 > >  There is a new version of XYZ available. View version 2.0 details.
 ***Automatic update is unavailable for this plugin.***
 >
 > By clicking the `View version 2.0 details` you'll see the plugin detail
 modal with the `Update now` button, which will fail with an error of
 `Update failed: Update package not available.`.

 Ugh, a bug. Needs fixing.

 > As a result, item 4 is not advisable, and the functionality mostly
 exists to support cases where the plugin is not hosted by WordPress.org
 and the update provider is not offering an update (but knows there's an
 update available).
 >
 > That leaves us with options 2 and 3.

 Right, in the above idea the API will be returning 3 until the pre-
 determined (percentage|threshold) number is reached, and then switch to
 return 2. Once manual override is implemented in core the API can include
 a flag to tell WP that there is an update that is part of a phased
 release, and the particular site was skipped. Thinking it would be okay
 for only new versions of WP to have this functionality.

 > I would suggest this feature needs to be limited to >10k active installs
 for plugin author simplicity. 10k is "too small" for usefulness in many
 respects, but choosing a larger count is likely to exclude some plugins
 that would benefit from it.

 Frankly my gut feeling was 50k active installs, going down to 20k after
 couple months, but I see your point :)

 > As a result, any phased rollouts for plugins **realistically** has a
 large lag time.

 Seems this will be true for plugins with less active installs. Plugins
 with lots of installs (>1m) will probably get a lot of manual updates too
 so the rollout will be quite fast.

 > I've graphed some initial thoughts of what I would expect as the types
 of rollouts buckets

 Sounds good but maybe can be considered as a later update/enhancement?
 Seems that the important functionality of phased releases can be achieved
 relatively simply, and buckets are not strictly necessary?

 > > Another thing that would be helpful for plugin authors here would be
 more analytics in the view to see how the rollout is progressing and such
 - perhaps in a simple dashboard.
 >
 > What data would you want to see here?
 > ...
 > What I think you're asking for is a progress meter

 Again, sounds really good but perhaps not in the initial release?

 > I'm still fairly concerned that we don't have any reasonable ways for
 authors to **really know** that there's a problem. I'd like to hear more
 from Plugin Authors on how we can address that, or if it's really
 something we're not going to be able to begin to solve until after this
 feature has been used for a while.

 Thinking that monitoring the forums after a new release is a fairly
 standard practice. Right, there can be better ways, but they all require
 the plugin's users to act. Maybe having a "Report a problem" button on
 each plugin row in the Plugins list table would help somewhat. That can
 also include the output from site health. But realistically this is a
 (separate) WP core problem.

-- 
Ticket URL: <https://meta.trac.wordpress.org/ticket/8009#comment:20>
Making WordPress.org <https://meta.trac.wordpress.org/>
Making WordPress.org