[wp-trac] [WordPress Trac] #50401: Tests: Move tests into Github Actions

[WordPress Trac]

#50401: Tests: Move tests into Github Actions
-------------------------------------+-----------------------------
 Reporter:  whyisjake                |       Owner:  desrosj
     Type:  enhancement              |      Status:  assigned
 Priority:  normal                   |   Milestone:  Future Release
Component:  Build/Test Tools         |     Version:
 Severity:  normal                   |  Resolution:
 Keywords:  has-patch needs-testing  |     Focuses:  performance
-------------------------------------+-----------------------------

Comment (by desrosj):

 Replying to [comment:9 desrosj]:

 > - The PHP compatibility ruleset explicitly specifies `./src/` as the
 only directory to run the checks on. However, `lint-action` has `.` hard
 coded when invoking `phpcs`. I was able to fix this by specifying
 `/tests/` as an exclude pattern within the ruleset.

 I've removed the `lint-action` from the workflows. While using maintained
 actions to setup and run the linting commands is nice, the PHPCS commands
 are only one line, so it's not unreasonable to hard code those instead of
 utilizing an action. Hard coding the commands also fixes the issue
 detailed above where it was impossible to lint just the `tests` directory
 with warnings enabled.

 The below issues I mentioned above are also fixed in the latest patch:

 > - The `PHP_FPM_UID` and `PHP_FPM_GID` environment variables need to be
 made dynamic.
 > - For some reason, failures are not always reported on the correct
 workflow. For example, if coding standards or compatibility checks fail,
 they frequently are reported on [https://github.com/desrosj/wordpress-
 develop/actions/runs/304876868 other random workflows run from the same
 commit].

 The failures not being reported on the correct workflow is due to an issue
 with how the GitHub checks API works. Checks are attached to commit SHAs,
 and the failure reports/annotations were being added to the newest
 workflow run for the  commit being tested. By running the commands
 separate form the `lint-action` action, we are able to annotate the commit
 [https://github.com/staabm/annotate-pull-request-from-checkstyle using a
 Checkstyle XML-report]. The `cs2pr` tool can be installed when PHP is set
 up earlier in the workflow (props @ocean90 for the help figuring that
 out).

 Also in the latest patch:
 - NodeJS is now installed before setting up caching for NPM. This ensures
 the [https://github.com/actions/cache/blob/main/examples.md#node---npm
 correct version of NPM is cached].
 - The events triggering builds have been refined. This will prevent every
 push on a PR AND the PR itself from running duplicate workflows. I also
 used expressions to try and scope workflows that should not be run on
 early branches. For example, the coding standards scans were not added
 until WP 5.1, and the PHP compatibility scans were not added until WP 5.5.
 - I changed the version of PHP that is configured within linting jobs from
 `latest` to `7.4`. This will prevent anything from breaking when `latest`
 becomes PHP 8 (just in case our tooling is not ready).

 This leaves two items to figure out. They are not blockers for committing
 these workflows, but they will need to be tackled eventually:
 - Slack notifications
 - Reading the desired version of NodeJS specified in the `.nvmrc` file.

 I played around with reading the `.nvmrc` file, but there are some issues.
 NVM supports aliases such as `lts/*` (which is the version specified in
 newer branches of WP), but the `actions/setup-node`
 [https://github.com/actions/setup-node/issues/26 does not] (and also
 [https://github.com/actions/setup-node/issues/32 does not support reading]
 `.nvmrc` files). This can be solved later on.

 I also did some rough estimating of speeds. It's hard to get 100% accurate
 comparisons because I have been changing the workflows as I go, but here's
 how things breakdown:
 - E2E tests: almost identical
 - PHP compatibility: Roughly 1/2 the time
 - JavaScript testing and linting/PHP linting: These are hard to compare
 because the QUnit tests were split into their own workflow, and the JSHint
 checks have been merged into a singular coding standards workflow with the
 PHP checks.
 - PHP 8-7.0: Between 3-5 minutes faster.
 - PHP 5.6: roughly the same.

-- 
Ticket URL: <https://core.trac.wordpress.org/ticket/50401#comment:11>
WordPress Trac <https://core.trac.wordpress.org/>
WordPress publishing platform