[wp-trac] [WordPress Trac] #30462: Test multiple supported database type and version combinations on GitHub Actions (was: Test multiple MySQL versions on GitHub Actions)

[WordPress Trac]

#30462: Test multiple supported database type and version combinations on GitHub
Actions
-------------------------------------------------+-----------------------
 Reporter:  pento                                |       Owner:  desrosj
     Type:  enhancement                          |      Status:  assigned
 Priority:  normal                               |   Milestone:  6.4
Component:  Build/Test Tools                     |     Version:
 Severity:  normal                               |  Resolution:
 Keywords:  has-patch 2nd-opinion needs-testing  |     Focuses:
-------------------------------------------------+-----------------------
Changes (by desrosj):

 * keywords:  has-patch => has-patch 2nd-opinion needs-testing
 * milestone:  Future Release => 6.4


Comment:

 I had some great discussions with @l1nuxjedi and @johnbillion at WordCamp
 Europe 2023 around this. I've finally had some time to take a look at this
 following that, and here's a breakdown.

 First, since I last commented above with versions:
 - PHP 8.0, 8.1, 8.2 were officially released.
 - WordPress has dropped support for PHP 5.6 (see [56141]).
 - MySQL 5.6 reached EOL (see [52420], [54069]) and is no longer supported.
 - Several versions of MariaDB are now marked as unmaintained (10.2 and
 10.3). 10.4 will be supported through June 2024.

 After chatting with @l1nuxjedi, these are the recommended
 [https://mariadb.com/kb/en/mariadb-server-release-dates/ versions of
 MariaDB] that we should be worried about testing against:
 - 10.4 (oldest stable release actively maintained)
 - 10.6 (previous long-term release, maintained through July 2026)
 - 10.11 (current long-ter stable release, maintained through February
 2028)
 - 11.0 (current stable release, maintained through June 2024)

 Paring this down from all actively maintained MariaDB versions and
 eliminating PHP 5.6 from the equation cuts down the testing strategy
 matrix quite a bit.

 I've explored a few options to tackle this, and have two PRs attached here
 to get feedback on.

 Option 1 - [https://github.com/WordPress/wordpress-develop/pull/5015
 PR-5015]: This option keeps everything to do with the PHPUnit workflow the
 same and just expands the strategy matrix. This is a cleaner diff, but
 does require a half dozen `exclude` statements to be added in order to
 avoid invalid combinations.

 Option 2 - [https://github.com/WordPress/wordpress-develop/pull/5016
 PR-5016]: This moves the actual testing aspect of the PHPUnit workflow to
 a callable one. This allows for two jobs with two unique strategy matrices
 specific to each database type, passing configuration details to the new
 callable workflow.

 I'm leaning towards option 2. I quite like how option 2 is displayed in
 the GHA interface (each job is tucked in an expandable element for the
 respective PHP versions) however it's not displayed the same on the
 "checks" screen for a PR (the PHP version is repeated). I also don't
 really love requiring 6 `exclude` statements. It's possible that could
 grow in the future as more versions are added. I'm indifferent about
 having a new callable workflow.

 These do result in a 5x increase in the number of jobs. However, I don't
 think it's a bad thing. I had considered having a smaller set of versions
 that we run on every commit, and a scheduled event that tested the full
 set of combinations. But after chatting with @johnbillion I don't think
 that's better. There's also room for improvement to make the tests run
 faster. They're currently creeping above 5 minutes per job, but it should
 be possible to cut that down.

 Looking to see what others think!

-- 
Ticket URL: <https://core.trac.wordpress.org/ticket/30462#comment:32>
WordPress Trac <https://core.trac.wordpress.org/>
WordPress publishing platform