[wp-trac] [WordPress Trac] #43395: Add Automated E2E Tests for Core Updates

[WordPress Trac]

#43395: Add Automated E2E Tests for Core Updates
-----------------------------+--------------------
 Reporter:  jorbin           |       Owner:
     Type:  defect (bug)     |      Status:  new
 Priority:  normal           |   Milestone:  4.9.5
Component:  Upgrade/Install  |     Version:
 Severity:  normal           |  Resolution:
 Keywords:  2nd-opinion      |     Focuses:
-----------------------------+--------------------

Comment (by netweb):

 In todays #core dev-chat a discussion arose on the feasibility of
 including ''new files'' in a ''minor'' release, e.g. 4.9.5, historically
 this has been avoided due to the how ''auto-updates'' work.


 *
 [https://wordpress.slack.com/archives/C02RQBWTW/p1520481343000204?thread_ts=1520470243.000286&cid=C02RQBWTW
 https://wordpress.slack.com/archives/C02RQBWTW/p1520481343000204]

 > dd32 Wrote:
 > ''I’d also just like to add, I’d love to see the core E2E update tests
 operating before any change to how we treat updates, purely because a lot
 of it is based on unknowns and the past 4 years of experiences.''

 * https://wordpress.slack.com/archives/C02RQBWTW/p1520470243000286

 > dd32 Wrote: re: 4.9.5 including new files:
 > - I’m not going to prevent that happening, but I’ll repeat my previous
 stance that automatic updates wasn’t designed with that in mind, and was
 specifically built at a time when minor releases would only ever be bugfix
 releases with no new files.
 >
 > - There were technical reasons why we only did minor releases and not
 major ones too - adding new files is effectively a major release when
 you’re looking at it from an upgrade point of view.
 >
 > - The largest failing case for autoupdates at present is
 `files_not_writable` where WordPress can’t modify existing files, let
 alone create new files (and this is after it’s already aborted because it
 knew it couldn’t write to files).
 >
 > - Adding new files has a chance that a %age of users may no longer
 receive security auto updates afterwards when the update fails.
 >
 > - Sites may also fatal error if new files are relied upon from another
 file and the autoupdate couldn’t create the files - we’ve never really
 tested the rollback-on-fail code as it’s only been triggered a handful of
 times.
 >
 > - There’s also some environments where if an autoupdate adds a file, the
 following user-initiated updates using FTP may fail as it’ll be unable to
 update the file - that’s one of the reasons we specified no-new-files.
 >
 > - If I had to guess, the last two use-cases would be in the region of
 0.5%~5% of users each.
 >
 > - It’s also worth reminding that in the event the autoupdate to 4.9.5
 fails, it’ll stay on 4.9.4 until the owner updates the site (much like
 we’ve got with 4.9.3 right now) we won’t be able to tell that site to
 update again in any manner, so even a 4.9.4.1 autoupdate would be ignored.
 >

 ----

 Further backgrount can be read (including some detatiled statsistics)
 here: https://make.wordpress.org/core/2017/03/11/continuing-inline-docs-
 improvements-adjacent-to-4-8/#comment-32248 , here's my quick extract of
 the key points:
 > Generally, update statistics have showed a clear correlation between
 failures and I/O operations. This is clearly seen in the difference
 between minor releases of different sizes, and even more pronounced when
 comparing those to a major release like 4.7 (with hundreds of files
 changed).
 >
 > Additionally, we run these updates using partial build packages. Every
 extra file adds a significant amount of KB to the package, which adds up
 pretty quickly. This not only stretches the wordpress.org load balancers
 (when suddenly millions of sites are updating within an hour), but also
 each individual site, which must download the ZIP, which takes time
 (partial builds made things, on many shared servers, go from minutes to
 seconds) and introduces lots of possibilities of download failures, and
 thus sites needing to retry later (and wait longer) to get patched.

--
Ticket URL: <https://core.trac.wordpress.org/ticket/43395#comment:8>
WordPress Trac <https://core.trac.wordpress.org/>
WordPress publishing platform