[wp-meta] [Making WordPress.org] #8009: Phased releases and roll-outs of plugins
Making WordPress.org
noreply at wordpress.org
Tue Jul 8 02:22:42 UTC 2025
#8009: Phased releases and roll-outs of plugins
------------------------------+---------------------
Reporter: matt | Owner: (none)
Type: enhancement | Status: new
Priority: normal | Milestone:
Component: Plugin Directory | Resolution:
Keywords: |
------------------------------+---------------------
Comment (by dd32):
In this comment, please assume: Site is running version 1.0 of a plugin,
version 2.0 is just released with a phased rollout, and if there's a bug
is found, version 2.1 is released.
After looking further into this, I ran into a few shortcomings that has
resulted in my suggestion in the earlier comments being not possible, or
with some caveats
Firstly:
**How to bucket / segment the sites.**
The most obvious way to segment the sites is to extract the sites domain
from the update-check HTTP user-agent, hash it, and convert it to a 0..100
number. That is the simplest and works relatively well for other places we
bucket sites on WordPress.org (eg: Core auto-updates throttles auto-update
availability during excessive network load)
There's a shortcoming here though, a site will always be in the same
bucket of sites - always at 8% for example.
Instead of hashing the domain, we could hash the entire user-agent header.
That would result in the bucket changing each time they update core. 8%
one time, 78% the next version. That would be ideal, as the user is not
always the "beta tester".
However, that means that sites that are in the first 1% are always getting
the first updates for all plugins using this update strategy.
The ideal solution is therefor to calculate the percentage as:
{{{
percent = convert_hash_to_percentage( hash( site-domain-name + plugin-slug
+ plugin-version ) )
}}}
which results in the percentage always changing, but consistent for any
specific plugin-update.
We'd have to exclude the WordPress version (and the entire user-agent
header) as percentages would otherwise change when core updates are
occuring. Including the plugin-version seems best to ensure a user isn't
always the canary, but does result in their position in the update queue
to change version-on-version, which causes an edge-case below.
> Perhaps having a threshold would be better? Something like 3-4k seems
sensible? This number would be enough to detect if something in the
release has gone really wrong, and fixes are needed urgently, and at the
same time will protect the bulk of the plugin's users from having the same
problems.
Using a count threshold **could** be better. However, we'd have to store
the number of sites we've offered each update to (and noting: We'd have to
store a site identifier, as WordPress may ask for an update check
literally every 5 minutes). Storing/Counting the number of sites that have
updated to a version wouldn't be relevant, as there could be 10x that
number who know about the update who are currently sitting on it until
their auto-update time comes around again (See the timing section later
on)
**With the way plugin updates are currently, it's an all-or-nothing
approach.**
WordPress currently only accepts a single plugin update payload, which
doesn't have any real ability to configure it.
My original intention here was to delay implementation of this into
WordPress 6.9, however, we still need to support WordPress 2.8 ~ 6.8
clients anyway so we have to take those into account. It would be
unexpected to a plugin author that this only applies for WordPress 6.9+
sites IMHO.
The major shortcoming of the existing WordPress plugin update check is
that we currently cannot provide different details for auto-updates vs
user-initiated updates.
We also cannot tell WordPress `"Here's an update that the user can
install, but DO NOT auto-update to it."`.
As a result, auto-updates-only is currently not possible with stable
versions of WordPress.
** What can we instruct WordPress to do? **
The WordPress.org plugin update API can return one of 4 states:
1. Plugin XYZ is not hosted on WordPress.org
2. Plugin XYZ is hosted on WordPress.org, and you're running the latest
version: 1.0
3. Plugin XYZ is hosted on WordPress.org, and you're running an outdated
version: 1.0. Version 2.0 is the latest.
4. Plugin XYZ is hosted on WordPress.org, you're running an outdated
version (1.0 vs 2.0) BUT automatic updates (both user-initiated and
background auto-updates) are not available for this plugin.
WordPress **needs** to know if the plugin is hosted on WordPress.org, so
we have to return one of 2, 3, or 4.
By returning 4, WordPress will display a notice such as the following:
> There is a new version of XYZ available. View version 2.0 details.
***Automatic update is unavailable for this plugin.***
By clicking the `View version 2.0 details` you'll see the plugin detail
modal with the `Update now` button, which will fail with an error of
`Update failed: Update package not available.`.
As a result, item 4 is not advisable, and the functionality mostly exists
to support cases where the plugin is not hosted by WordPress.org and the
update provider is not offering an update (but knows there's an update
available).
That leaves us with options 2 and 3.
The simplest option forward here is, when 2.0 is released that we say:
- (2) Site not included (yet): You're running version 1.0, there's no
updates available
- (2) Site not included (yet): You're running version 0.5, there's no
updates available ***(It's not the latest version, but because they're not
included in the most recent update group yet, it's all we can tell them -
we can't really store previous version data here, without causing
additional awkward edge-cases)***
- (3) Site included in phased rollout: Hey version 2.0 of this plugin is
out! Update now!
That's not a great user experience, as a user is unable to update a site
even though they know the update is available on another site. But we
don't (in WordPress 6.8) have another choice.
However, edge-cases are rough here, Let's say at the 25% rollout point a
bug is discovered in the plugin, what does the author do?
- Rollback: The author sets 1.0 as the stable release again.
- Fixed Release: The author releases 2.1 to fix it.
In the rollback case, those running v2.0 ***will not be downgraded*** (and
IMHO: shouldn't be), and if the author takes a week to move to fixing it,
they'll continue to run the buggy version. That's highly likely acceptable
to us.
In the fixed release case, assuming they use the same rollout strategy.
Going back to the percentage above and the canary case.. that means that
the bucket of users who receive the update first will be different to
those who updated before - in other words, Site ABC was the FIRST site to
update to version 2.0, but now there's a chance they'll be the millionth
site to update OR the LAST to update to version 2.1.
I see this as just an acceptable edge-case. We could do logic like "If
they received the 2.0 update, update to 2.1 immediately" but that would
require storing additional data about the site and what it had been
offered.
Another edge-case here is that if the site was running version 0.5 of the
plugin, and saw an update to 1.0 is available, having the site attempt to
update to 1.0 MAY fail if version 2.0 is released at the same time. This
is due to race conditions between the WordPress.org API, the transients in
WordPress, and the way the user-interface uses that data.
This is just an edge-case I'm documenting here, that I don't believe will
realistically ever be an issue.
** Install Count **
Phased updates using the hashing method of bucketing is likely to result
in at least one site always being in a bucket, however, smaller plugins
will inherently have less sites per bucket. Combine this with the fact
that many sites do NOT update their plugins, and do not have auto-updates
for plugins enabled, means it could be a while for smaller plugins to get
an update.
I would suggest this feature needs to be limited to >10k active installs
for plugin author simplicity. 10k is "too small" for usefulness in many
respects, but choosing a larger count is likely to exclude some plugins
that would benefit from it.
** Update Timing **
I've modeled a few different strategies for update availability, but
there's a few major kinks in how WordPress plugin update-checks operate.
WordPress has a decoupled notion of how auto-updates operate, and how that
interacts with WordPress knowing about a plugin update.
Consider this:
- WordPress core/themes/plugin update checks run twice daily
- WordPress auto-updates only run during the core update check
When you visit the WordPress plugins page, it might find out about the
plugin update immediately (as it just performed the update check) but it
won't auto-update the plugin until the twicedaily core update check runs.
In those situations you'd see something like:
> There is a new version of Plugin XYZ available. View version 2.0
details or update now.
> Disable auto-updates
> Automatic update scheduled in 11 hours.
As a result, any phased rollouts for plugins **realistically** has a large
lag time. This increases the overall length of time that updates need to
be released at. We're not talking "5% in the first hour, and increase 5%
every hour after that" We're talking "5% in the first 12 hours, and 5%
every 12 hours thereafter".
We could:
- Schedule auto-updates to run immediately once it knows there's an
update (of any kind) available and the next scheduled auto-update is not
for hours.
I've graphed some initial thoughts of what I would expect as the types of
rollouts buckets, I personally only feel that the `cautious (5d)` and
`slow (3.5d)` strategies are worthwhile when you consider the lag above.
[[Image(https://cldup.com/GXD81vmE4E.png)]]
> Another thing that would be helpful for plugin authors here would be
more analytics in the view to see how the rollout is progressing and such
- perhaps in a simple dashboard.
What data would you want to see here? Noting that we can't realistically
provide the number of sites updated (or a total count of sites), we could
show "60% are running this version" but that's not useful if the other 40%
are running a version 10x older. "5x more people are using this than
LAST_RELEASE" also not super useful IMHO.
What I think you're asking for is a progress meter, but we don't really
know what the upper-bound of that is, so we can't really calculate the
progress. The upper bound is NOT the active install count.
** Feedback to author **
Finally, having a [https://en.wiktionary.org/wiki/canary_in_a_coal_mine
canary-in-the-coalmine] is not useful if the workers are not watching the
canary or if the workers are in a different mine all together.
I alluded to this earlier, and elsewhere suggested that reviews/ratings
could form part of this.
> Matt: If there were a way to rate and review without leaving wp-admin,
that would also be amazing for plugin devs, and lower friction than our
current flow.
Bad reviews are far more likely to come from an intentional change in
behaviour than bugs IMHO, where a phased rollout would likely not prevent
it. As such, i've split this into #8031 as it'll also require some
significant work on how forums reviews work, how WordPress.org account
registration works, and how spam would be dealt with.
But to focus on my main concern here: If a plugin is rolled out to 5% of
users, and it's a bug that only 10% of users run into, then that's 0.5% of
users who are both affected and updated, and then consider that maybe only
one-in-ten would actually go to the effort of reporting it, then we're at
a tiny likelyhood of issues being reported for some plugins. For multi-
million-install plugins this isn't likely an issue as the issue would
likely be reported fast.
But to use my experience of monitoring the forums, when a plugin releases
a **slightly bad** update it can generate maybe 3 threads on the forums
with a dozen people interacting with it.. and that's when it takes the
author several days to respond to the issues.
I'm still fairly concerned that we don't have any reasonable ways for
authors to **really know** that there's a problem. I'd like to hear more
from Plugin Authors on how we can address that, or if it's really
something we're not going to be able to begin to solve until after this
feature has been used for a while.
--
Ticket URL: <https://meta.trac.wordpress.org/ticket/8009#comment:11>
Making WordPress.org <https://meta.trac.wordpress.org/>
Making WordPress.org
More information about the wp-meta
mailing list