[wp-trac] [WordPress Trac] #34694: Facilitate automated testing in context of pull requests and diffs

[WordPress Trac]

#34694: Facilitate automated testing in context of pull requests and diffs
------------------------------+----------------------------
 Reporter:  westonruter       |      Owner:
     Type:  enhancement       |     Status:  new
 Priority:  normal            |  Milestone:  Future Release
Component:  Build/Test Tools  |    Version:
 Severity:  normal            |   Keywords:
  Focuses:                    |
------------------------------+----------------------------
 Using GitHub for core development has been a wishlist item for awhile now.
 Individual feature plugins have been increasingly developed on GitHub
 (e.g. Widget Customizer, WP-API). Going an extra step, the Twenty Sixteen
 theme only exists on GitHub and isn't even committed to `trunk`. So while
 feature plugins and themes are often being developed primarily on GitHub,
 other work on Core is generally not (although
 [https://make.wordpress.org/core/2015/09/21/wp-rest-api-merge-proposal/
 this may change]).

 While there is no official wordpress-develop Git repo on GitHub (yet),
 anyone can clone the [https://make.wordpress.org/core/2014/01/15/git-
 mirrors-for-wordpress/ wordpress-develop Git repo] and push it to GitHub.
 Teams working on various components can then work in the context of those
 GitHub clones, and then when work is complete a committer can apply the
 patch to SVN. Travis CI is already configured in the wordpress-develop
 repo, and so anyone who clones the repo to GitHub can automatically turn
 on Travis CI to start getting automated build checks for each commit.

 Work on GitHub is usually done in the context of pull requests, and Travis
 CI can be especially useful when running in response to a pull request
 being opened or a commit pushed to a feature branch with an existing pull
 request. As reported in #30017, many automated tests are unnecessarily
 slow. This is in large part because with each commit, _all_ checks are
 being made across all files in the project, irrespective of whether they
 were recently modified or not.

 When Travis CI does a build in the context of a pull request, it makes
 available the branch checked out and the branch being merged into, and
 with these two refs in hand, we can gather a list of the files (and
 patches) that were specifically changed. When we have these, we can
 optimize Travis CI to only run the checks that are relevant to the changes
 in the feature branch. For instance, if no PHP files were changed, then
 all of the jobs that run PHPUnit tests can be cancelled. If no JS files
 were modified, then the JSHint checks can be skipped.

 It can be painful to introduce new automated checks to WordPress because
 they can add a lot of noise for automated tests and fixing requires a
 touching a lot of files, possibly invalidating many pending patches. When
 Travis runs in the context of a pull request (or when automated tests are
 run when `trunk`/`master` is not checked out), then again, there is a diff
 available for the specific changes. We can introduce PHP_CodeSniffer for
 core (#30153) when Travis runs in in the context of a pull request, and it
 can skip reporting errors on any lines that aren't modified in the commit.
 This is likewise how JSCS linting can be added (#31823).

 For existing code which causes Travis CI to only report coding standard
 violations to lines changed, see https://github.com/xwp/wp-dev-
 lib/blob/master/travis.script.sh when `$LIMIT_TRAVIS_PR_CHECK_SCOPE` is
 `patches`.

 For commits pushed to `master` on Travis CI (outside the context of a pull
 request), it may make sense to configure Travis to check for changes made
 since the last major release.

--
Ticket URL: <https://core.trac.wordpress.org/ticket/34694>
WordPress Trac <https://core.trac.wordpress.org/>
WordPress publishing platform