[wp-trac] [WordPress Trac] #52645: Split slow tests into their own job on PHP 5.6 and run them in parallel

[WordPress Trac]

#52645: Split slow tests into their own job on PHP 5.6 and run them in parallel
------------------------------+-----------------------------
 Reporter:  johnbillion       |      Owner:  (none)
     Type:  task (blessed)    |     Status:  new
 Priority:  normal            |  Milestone:  Awaiting Review
Component:  Build/Test Tools  |    Version:
 Severity:  normal            |   Keywords:  needs-patch
  Focuses:                    |
------------------------------+-----------------------------
 The test suite takes ~16 minutes to run on GitHub Actions, which was
 recently reduced from ~26 minutes in #52548. The approach taken was to
 split up the jobs so each one runs faster, and then run them in parallel
 so the overall duration is reduced.

 The PHPUnit tests on PHP 5.6 are the slowest jobs by far, taking ~12
 minutes compared to 5 to 7 minutes for all other PHP versions. If we can
 reduce the time that the PHP 5.6 runs take then we reduce the overall
 duration.

 I propose that for the PHP 5.6 runs specifically, we split the tests in
 half by selecting the slowest test groups whose combined time is roughly
 half the time of the whole test run, running them in a separate job, and
 excluding them from the main job. As the jobs will run in parallel this
 will reduce the overall duration.

 After [https://github.com/WordPress/wordpress-
 develop/actions/runs/590468981 a bunch of testing] I've determined that
 the `external-http`, `media`, and `restapi` groups combined take roughly
 the same time as all the other tests combined.

 * It's known that the `external-http` tests are slow which is why they are
 run separately by default
 * The `media` tests are especially slow on PHP 5.6
 * The `restapi` test group contains a large number of tests and therefore
 consumes a large proportion of the total duration

 Running these groups separately [https://github.com/WordPress/wordpress-
 develop/actions/runs/597141794 reduces the overall duration by around 4
 minutes].

 Combining this group separation with
 [https://core.trac.wordpress.org/ticket/51734 running the tests from src
 instead of build] means [https://github.com/WordPress/wordpress-
 develop/actions/runs/597293789 the overall duration dips below 9 minutes].

 So, why not do the same for PHP versions other than 5.6? The answer is
 that for all other PHP versions there is no net gain because the time
 saved by splitting the tests in half is less than the time taken for the
 job to be set up and run. Only with PHP 5.6 does the time saving outweigh
 the time taken to set up the job.

-- 
Ticket URL: <https://core.trac.wordpress.org/ticket/52645>
WordPress Trac <https://core.trac.wordpress.org/>
WordPress publishing platform