[wp-trac] [WordPress Trac] #58977: Consider adopting workflows for testing build packages

[WordPress Trac]

#58977: Consider adopting workflows for testing build packages
------------------------------+-----------------------
 Reporter:  costdev           |       Owner:  costdev
     Type:  enhancement       |      Status:  assigned
 Priority:  normal            |   Milestone:  6.4
Component:  Build/Test Tools  |     Version:
 Severity:  normal            |  Resolution:
 Keywords:  has-patch commit  |     Focuses:
------------------------------+-----------------------
Changes (by desrosj):

 * keywords:  needs-patch => has-patch commit


Comment:

 I've been working with @costdev quite a bit over on the
 [https://github.com/WordPress/wordpress-upgrade-tester/ WordPress
 /wordpress-upgrade-tester repository] where we moved his POC.

 I think this is in a great state. It currently:
 - Tests updating all branches of WordPress that receive security updates
 to the version being tested.
 - Tests updating with all currently supported PHP versions (where
 possible).
 - Tests updating with all currently supported MySQL versions.

 Let's get this in to see how it goes throughout the 6.4 beta/RC period
 leading up to final release.

 For now, it will only run if manually triggered. I've checked that this is
 possible (and it is), but in the future when the WP.org server creates the
 release package for a specific version, it should make a request to GitHub
 with the version number being packaged to trigger this workflow
 automatically.

 Here's a list of remaining tasks that are not blockers, but need to be
 explored further (not necessarily in this ticket):
 - Perform some actual test assertions. Currently, the workflow only checks
 that WP-CLI does not encounter a failure when attempting to update. It
 would be good to also have some tests that perform some basic tasks to
 confirm things look good and actually work.
 - When testing minor releases for older versions, we should only test up
 to that version. For example, if 5.0.10 is being released, we shouldn't
 test updating 5.1.x to 5.0.10. Only WP versions lower than the current
 version should be tested.
 - When testing minor releases for older versions, we should test the
 versions of PHP that were supported for that version. For example, if
 we're releasing a [https://make.wordpress.org/core/handbook/references
 /php-compatibility-and-wordpress-versions/ 4.9 minor version, we should
 test PHP 5.2-5.6, and 7.0-7.2]. This also applies for database versions.
 - Ideally, the workflow should also perform MariaDB testing. After
 [56439], MariaDB is tested by default for all commits.

 One last call out is that the workflow takes a bit of time to report
 success. However, the jobs in the workflow itself run surprisingly fast.
 The time that it actually takes to run all of the associated jobs can be
 calculated by monitoring the GitHub Action Runner administration page when
 the workflow starts. The jobs are all completed within 5-8 minutes, but
 the delay seems to be with the product UI receiving and displaying job
 details from the 650+ runners, and takes double to triple that amount of
 time.

 During the release process, this could be a pain, but that usually takes
 around that same amount of time. But this workflow is not meant to replace
 manual testing, but rather serve as an additional sanity check to catch
 some additional potential issues. I've reached out to GitHub about this,
 and we can continue to explore ways to speed this up, but I think it will
 need to be fixed on their end.

-- 
Ticket URL: <https://core.trac.wordpress.org/ticket/58977#comment:10>
WordPress Trac <https://core.trac.wordpress.org/>
WordPress publishing platform