[wp-trac] [WordPress Trac] #28301: Improved Travis CI Performance and Configuration

[WordPress Trac]

#28301: Improved Travis CI Performance and Configuration
------------------------------+-----------------------------
 Reporter:  netweb            |      Owner:
     Type:  enhancement       |     Status:  new
 Priority:  normal            |  Milestone:  Awaiting Review
Component:  Build/Test Tools  |    Version:  trunk
 Severity:  normal            |   Keywords:
  Focuses:  performance       |
------------------------------+-----------------------------
 Based on this comment by Jorbin: ticket:26446#comment:2
 > I would love to figure out a way to not have to run jshint for each of
 the configurations since you are correct that it doesn't need to be run
 across multiple build environments. I'm not sure if Travis supports that,
 but I imagine we can code around that if you really think it is a show
 stopper. Of course, as you point out we are only adding ~16 seconds to the
 travis build.

 The attached patch does the above and a few more things:

 * Switches to a using a 'build matrix' with explicitly defined PHP version
 and environment variable for each build job, each environment variable is
 a new Grunt task.

 * Existing build jobs no longer include the `qunit:compiled` or `jshint`
 tests, they each have their own dedicated build job using PHP 5.5 and do
 NOT include PHPUnit testing, MySQL database creation, or SVN checkout of
 WordPress Importer. ''(No specific reason I chose PHP 5.5, I had to pick
 one is all)''

 * The job build order is prioritized so that the `qunit` and `javascript`
 tests are run first, both of these tasks have in testing remained under
 120 seconds and 90 seconds respectively.

 * The addition of the `fast_finish: true` flag in the build matrix will
 mark the build `passed` or `errored` as soon as a single build job has
 failed. Previously the build status was not reported until ALL tests had
 completed regardless of status. In theory if a `jshint` test failed
 previously you would not know for ~30 minutes, this should now report in
 under two minutes once the build job has begun.

 * Moves the 'build configuration' to `before_install` which includes only
 if the build job is `travis:phpunit` the MySQL database creation, and
 WordPress Importer SVN checkout. When a failure occurs in this section the
 build is marked as `errored` which is a more accurate description in that
 the environment we are setting up has failed and not the actual WordPress
 tests we are testing.

 * Introduces and/or modifies existing Grunt test tasks for Travis CI
 testing, this allows each task to be targeted via the new environment
 variables in the build matrix called now by `script: grunt $WP_TRAVISCI`
 rather than the existing `grunt travis` command.
  * `grunt.registerTask('travis:js', 'Runs Javascript TravisCI tasks.',
 'jshint');`
  * `grunt.registerTask('travis:phpunit', 'Runs PHPUnit TravisCI tasks.',
 'phpunit');`
  * `grunt.registerTask('travis:qunit', 'Runs QUnit TravisCI tasks.',
 'qunit:compiled');`

 * Introduces PHP 5.6 and HHVM testing into the build matrix, both of these
 are ''allowed to fail'' in that they will not be included in the overall
 build job `failed`, `errored`, or `passed` build status. This allows
 WordPress testing to play nice with these alpha/beta version of PHP/HHVM
 until stable and officially released.

 Build Matrix Summary
 ||Duration||ENV||PHP||Allowed Failures||
 ||1 min 45 sec||WP_TRAVISCI=travis:qunit||5.5||No||
 ||1 min 10 sec||WP_TRAVISCI=travis:js||5.5||No||
 ||5 min 2 sec||WP_TRAVISCI=travis:phpunit||5.2||No||
 ||7 min 46 sec||WP_TRAVISCI=travis:phpunit||5.3||No||
 ||8 min 13 sec||WP_TRAVISCI=travis:phpunit||5.4||No||
 ||9 min 5 sec||WP_TRAVISCI=travis:phpunit||5.5||No||
 ||7 min 43 sec||WP_TRAVISCI=travis:phpunit||5.6||Yes||
 ||1 min 22 sec||WP_TRAVISCI=travis:phpunit||hhvm||Yes||

 A couple of the Travis CI tests from my fork for this patch:
 https://travis-ci.org/ntwb/wordpress/builds/25441345 <- A forced `errored`
 build
 https://travis-ci.org/ntwb/wordpress/builds/25495226 <- The build this
 patch was submitted against

--
Ticket URL: <https://core.trac.wordpress.org/ticket/28301>
WordPress Trac <https://core.trac.wordpress.org/>
WordPress publishing platform