[wp-trac] [WordPress Trac] #34694: Facilitate automated testing in context of pull requests and diffs

[WordPress Trac]

#34694: Facilitate automated testing in context of pull requests and diffs
------------------------------+-----------------------------
 Reporter:  westonruter       |       Owner:
     Type:  enhancement       |      Status:  new
 Priority:  normal            |   Milestone:  Future Release
Component:  Build/Test Tools  |     Version:
 Severity:  normal            |  Resolution:
 Keywords:                    |     Focuses:
------------------------------+-----------------------------
Description changed by westonruter:

Old description:

> Using GitHub for core development has been a wishlist item for awhile
> now. Individual feature plugins have been increasingly developed on
> GitHub (e.g. Widget Customizer, WP-API). Going an extra step, the Twenty
> Sixteen theme only exists on GitHub and isn't even committed to `trunk`.
> So while feature plugins and themes are often being developed primarily
> on GitHub, other work on Core is generally not (although
> [https://make.wordpress.org/core/2015/09/21/wp-rest-api-merge-proposal/
> this may change]).
>
> While there is no official wordpress-develop Git repo on GitHub (yet),
> anyone can clone the [https://make.wordpress.org/core/2014/01/15/git-
> mirrors-for-wordpress/ wordpress-develop Git repo] and push it to GitHub.
> Teams working on various components can then work in the context of those
> GitHub clones, and then when work is complete a committer can apply the
> patch to SVN. Travis CI is already configured in the wordpress-develop
> repo, and so anyone who clones the repo to GitHub can automatically turn
> on Travis CI to start getting automated build checks for each commit.
>
> Work on GitHub is usually done in the context of pull requests, and
> Travis CI can be especially useful when running in response to a pull
> request being opened or a commit pushed to a feature branch with an
> existing pull request. As reported in #30017, many automated tests are
> unnecessarily slow. This is in large part because with each commit, _all_
> checks are being made across all files in the project, irrespective of
> whether they were recently modified or not.
>
> When Travis CI does a build in the context of a pull request, it makes
> available the branch checked out and the branch being merged into, and
> with these two refs in hand, we can gather a list of the files (and
> patches) that were specifically changed. When we have these, we can
> optimize Travis CI to only run the checks that are relevant to the
> changes in the feature branch. For instance, if no PHP files were
> changed, then all of the jobs that run PHPUnit tests can be cancelled. If
> no JS files were modified, then the JSHint checks can be skipped.
>
> It can be painful to introduce new automated checks to WordPress because
> they can add a lot of noise for automated tests and fixing requires a
> touching a lot of files, possibly invalidating many pending patches. When
> Travis runs in the context of a pull request (or when automated tests are
> run when `trunk`/`master` is not checked out), then again, there is a
> diff available for the specific changes. We can introduce PHP_CodeSniffer
> for core (#30153) when Travis runs in in the context of a pull request,
> and it can skip reporting errors on any lines that aren't modified in the
> commit. This is likewise how JSCS linting can be added (#31823).
>
> For existing code which causes Travis CI to only report coding standard
> violations to lines changed, see https://github.com/xwp/wp-dev-
> lib/blob/master/travis.script.sh when `$LIMIT_TRAVIS_PR_CHECK_SCOPE` is
> `patches`.
>
> For commits pushed to `master` on Travis CI (outside the context of a
> pull request), it may make sense to configure Travis to check for changes
> made since the last major release.

New description:

 Using GitHub for core development has been a wishlist item for awhile now.
 Individual feature plugins have been increasingly developed on GitHub
 (e.g. Widget Customizer, WP-API). Going an extra step, the Twenty Sixteen
 theme only exists on GitHub and isn't even committed to `trunk`. So while
 feature plugins and themes are often being developed primarily on GitHub,
 other work on Core is generally not (although
 [https://make.wordpress.org/core/2015/09/21/wp-rest-api-merge-proposal/
 this may change]).

 While there is no official wordpress-develop Git repo on GitHub (yet),
 anyone can clone the [https://make.wordpress.org/core/2014/01/15/git-
 mirrors-for-wordpress/ wordpress-develop Git repo] and push it to GitHub.
 Teams working on various components can then work in the context of those
 GitHub clones, and then when work is complete a committer can apply the
 patch to SVN. Travis CI is already configured in the wordpress-develop
 repo, and so anyone who clones the repo to GitHub can automatically turn
 on Travis CI to start getting automated build checks for each commit.

 Work on GitHub is usually done in the context of pull requests, and Travis
 CI can be especially useful when running in response to a pull request
 being opened or a commit pushed to a feature branch with an existing pull
 request. As reported in #30017, many automated tests are unnecessarily
 slow. This is in large part because with each commit, _all_ checks are
 being made across all files in the project, irrespective of whether they
 were recently modified or not.

 When Travis CI does a build in the context of a pull request, it makes
 available the branch checked out and the branch being merged into, and
 with these two refs in hand, we can gather a list of the files (and
 patches) that were specifically changed. When we have these, we can
 optimize Travis CI to only run the checks that are relevant to the changes
 in the feature branch. For instance, if no PHP files were changed, then
 all of the jobs that run PHPUnit tests can be cancelled. If no JS files
 were modified, then the JSHint checks can be skipped.

 It can be [https://make.wordpress.org/core/2013/11/13/finding-fixing-
 javascript-errors-with-jshint/ painful] to introduce new automated checks
 to WordPress because they can add a lot of noise for automated tests and
 fixing requires a touching a lot of files, possibly invalidating many
 pending patches. When Travis runs in the context of a pull request (or
 when automated tests are run when `trunk`/`master` is not checked out),
 then again, there is a diff available for the specific changes. We can
 introduce PHP_CodeSniffer for core (#30153) when Travis runs in in the
 context of a pull request, and it can skip reporting errors on any lines
 that aren't modified in the commit. This is likewise how JSCS linting can
 be added (#31823).

 For existing code which causes Travis CI to only report coding standard
 violations to lines changed, see https://github.com/xwp/wp-dev-
 lib/blob/master/travis.script.sh when `$LIMIT_TRAVIS_PR_CHECK_SCOPE` is
 `patches`.

 For commits pushed to `master` on Travis CI (outside the context of a pull
 request), it may make sense to configure Travis to check for changes made
 since the last major release.

--

--
Ticket URL: <https://core.trac.wordpress.org/ticket/34694#comment:1>
WordPress Trac <https://core.trac.wordpress.org/>
WordPress publishing platform