[wp-trac] [WordPress Trac] #43395: Add Automated E2E Tests for Core Updates

[WordPress Trac]

#43395: Add Automated E2E Tests for Core Updates
-----------------------------+--------------------
 Reporter:  jorbin           |       Owner:
     Type:  defect (bug)     |      Status:  new
 Priority:  normal           |   Milestone:  4.9.5
Component:  Upgrade/Install  |     Version:
 Severity:  normal           |  Resolution:
 Keywords:  2nd-opinion      |     Focuses:
-----------------------------+--------------------

Comment (by dd32):

 After speaking with a few people, here's the list I've come up with of a
 MVP and also a huge stretch goal of various matrix options which we
 can/should explore.
 The matrix below is huge and I'm sure I've missed a few angles, some
 things may not make sense to run together, but at the end of the day the
 WordPress upgrade routines run in a huge number of varied environments,
 and previously most of the testing has had to be done manually in the past
 due to lack of tooling available to automate it.

 A lot of what is presented below is just a brain dump of the various
 branches of code we have, some will make sense to run a full upgrade with,
 some probably only need some basic unit testing (`Variants of file system
 paths` would benefit from a small handful of phpunit tests run against
 core in a docker instance for example, but some should maybe get a full
 upgrade run).

 MVP:
  - Running an update of trunk (HEAD/master) to the latest nightly via
 various trigger methods (See `Various methods of updates` below). Trust
 WordPress when it says it completed successfully.

 More detailed ideas and stretch goals:
  - Running upgrades in Docker locally & via Travis CI seems to be the best
 method. Docker Compose & custom published DockerFiles should help there
  - Testing various types of upgrade
   - Minor x.y.Z => x.y.(Z+1) - With the expectation of it also failing
 under certain circumstances
   - Major x.Y => x.(Y+1)
   - Dev nightly (Trunk), Possibly Stable branch nightly while we're at it.
  - Various methods of updates:
   - Background Update (Triggered by `wp_version_check()` running in a cron
 environment)
   - Background Update (Triggered by calling `wp_maybe_auto_update()` - as
 some hosts do)
   - Via wp-cli (Depending on if it differs from the above methods?)
   - Manual updates through the UI - Clicking the button (Stretch goal of
 using a modern fake-browser)
  - Variants of filesystem method access
   - `direct` - expect that PHP owns and can modify all files - MVP
   - `direct_gw` - expect that PHP can modify own files, but not create
 them. PHP user is in the same group as the file owner. Should only be
 possible to do Minor updates
   - `ftp` - Have files be RO except via FTP.
    - Variant: FTP Extension
    - Variant: FTP "Sockets" (php ftp extension not available)
    - Use different FTP servers in different configurations. ProFTPD,
 PureFTPd and vsftpd - possibly one configured as a windows-ftp server
 (Yes, some Windows FTP servers have a different `ls` output)
  - Variants of file system paths
   - Nested installs
   - `.svn`/`.git` directories at various levels to block updates (We can
 probably relax that though)
   - PHP `__DIR__` not matching FTP paths
    - FTP may be `chroot()`'d presenting `/users/dd32/domain/xyz.com` as
 `/domain/xyz.com`
    - May rewrite paths completely, presenting `/home/dd32/domain/xyz.com`
 as `/users/1234/xyz.com`
  - Verify that WordPress actually upgraded successfully, and not just that
 WordPress said it completed successfully.
  - Verify WordPress respects the various update filters and constants for
 how it updates
   - `WP_AUTO_UPDATE_CORE`
   - `AUTOMATIC_UPDATER_DISABLED`
   - `CORE_UPGRADE_SKIP_NEW_BUNDLED`, etc

 Random ideas:
  - Dockerise the WordPress.org API and/or use a mu-plugin to force
 WordPress to see the API response we want it to (So it doesn't hit the
 WodPress.org api at all). That would allow for easier testing of
 `major`/`minor` updates.
  - Fetch zips locally that we create, to allow us to create a "fake"
 release zip. For example, current stable is WordPress 5.0.0, we create a
 `WordPress 5.5.0` zip as a manner to test that major updates can succeed
 from trunk to next-trunk.

 I don't have my full notes with me right now, but if I've missed anything
 major i'll update this later in the week when I'm back in front of them.
 I hope this is useful to someone else as well to kind of give some
 direction on what kinds of things we need to test here.

 I've personally run into a few brick walls while trying to get a basic MVP
 of this working, due to my limited experience with docker and related
 tooling, there's a great amount of iterative work which could be done once
 a MVP is running though. While I'll personally keep working on attempting
 this when I can, if there's anyone else who has the available time and
 knowledge to scaffold something basic up, that'd be a great start for the
 community.

--
Ticket URL: <https://core.trac.wordpress.org/ticket/43395#comment:4>
WordPress Trac <https://core.trac.wordpress.org/>
WordPress publishing platform