[wp-meta] [Making WordPress.org] #3760: Reliability issues with Travis theme download tests for WP-CLI

[Making WordPress.org]

#3760: Reliability issues with Travis theme download tests for WP-CLI
-----------------------------+--------------------
 Reporter:  schlessera       |      Owner:  (none)
     Type:  defect           |     Status:  new
 Priority:  normal           |  Milestone:
Component:  Theme Directory  |   Keywords:
-----------------------------+--------------------
 We are having constant reliability issues with our functional tests in WP-
 CLI, because the Travis environment seems to get a lot of random error
 from the .org infrastructure.

 Let me detail a specific example here.

 In the functional tests for the `wp-cli/extension-command` (which includes
 plugin and theme commands), we have the following failure:

 https://travis-ci.org/wp-cli/automated-tests/jobs/414224296#L2018-L2049

 {{{
 Couldn't find 'twentytwelve' in the WordPress.org theme directory.
 }}}

 This is the message we are getting from the .org servers. The command to
 be tested was the following:

 {{{
 wp theme install --force twentytwelve --version=1.0
 }}}

 I never managed to get this to fail when running it manually. However, in
 our functional tests, it often fails. And the results are the same:

 https://travis-ci.org/wp-cli/automated-tests/jobs/414224296#L2170-L2201

 https://travis-ci.org/wp-cli/automated-tests/jobs/414224296#L2322-L2353

 Given that the command is used and fails 3x for each run, this makes it 9
 consecutive tries to download that theme that failed.

 This makes me believe that there might be problems with the way Travis CI
 connects to the servers, and I would be grateful for any help at getting
 to the bottom of this.

 Build system information for that environment can be found starting from
 here: https://travis-ci.org/wp-cli/automated-tests/jobs/414224296#L6

 The IP addresses that Travis can use are the following:
 {{{
 147.75.192.163
 34.233.56.198
 34.234.4.53
 52.3.55.28
 52.45.185.117
 52.45.220.64
 52.54.31.11
 52.54.40.118
 54.208.31.17
 }}}

-- 
Ticket URL: <https://meta.trac.wordpress.org/ticket/3760>
Making WordPress.org <https://meta.trac.wordpress.org/>
Making WordPress.org