[wp-trac] [WordPress Trac] #48145: Random PHP test failures

WordPress Trac noreply at wordpress.org
Mon Nov 11 21:17:44 UTC 2019


#48145: Random PHP test failures
------------------------------+---------------------
 Reporter:  SergeyBiryukov    |       Owner:  (none)
     Type:  defect (bug)      |      Status:  new
 Priority:  low               |   Milestone:  5.4
Component:  Build/Test Tools  |     Version:
 Severity:  normal            |  Resolution:
 Keywords:  needs-patch       |     Focuses:
------------------------------+---------------------
Description changed by SergeyBiryukov:

Old description:

> Every few commits, there's now a random PHP 7.1 test failure on Travis,
> mostly off-by-one errors unrelated to the preceding commit. Started about
> 1 month ago, after August 20th.
>
> Seems like either PHP 7.1 tests are for some reason more prone to random
> errors than other versions, or it's just more prominent now that random
> date/time failures from #45821 are fixed.
>
> Creating a ticket just for reference, any ideas as to why this happens
> appreciated.
>
> Some examples:
> {{{
> 1) WP_Test_REST_Posts_Controller::test_get_items_search_query
> Failed asserting that 6 matches expected 7.
> /var/www/tests/phpunit/tests/rest-api/rest-posts-controller.php:459
> }}}
> {{{
> 1) WP_Test_REST_Posts_Controller::test_get_items_pagination_headers
> Failed asserting that 49 matches expected 50.
> /var/www/tests/phpunit/tests/rest-api/rest-posts-controller.php:1157
> }}}
> {{{
> 1) Tests_Canonical_PageOnFront::test with data set #3 ('/?page=1', '/')
> Failed asserting that two strings are equal.
> --- Expected
> +++ Actual
> @@ @@
> -'/'
> +'/?page=1'
> /var/www/tests/phpunit/includes/testcase-canonical.php:268
> /var/www/tests/phpunit/tests/canonical/pageOnFront.php:39
>
> 2)
> Tests_Query_MetaQuery::test_meta_query_single_query_compare_arithmetic_comparisons
> Failed asserting that two arrays are equal.
> --- Expected
> +++ Actual
> @@ @@
>  Array (
> -    0 => 2319
>  )
> /var/www/tests/phpunit/includes/abstract-testcase.php:658
> /var/www/tests/phpunit/tests/query/metaQuery.php:162
> }}}
> {{{
> 1) Tests_Query::test_tag_querystring_single_term
> Failed asserting that two arrays are equal.
> --- Expected
> +++ Actual
> @@ @@
>  Array (
>      0 => 2022
> -    1 => 2023
>  )
> /var/www/tests/phpunit/includes/abstract-testcase.php:658
> /var/www/tests/phpunit/tests/query.php:321
> }}}
> {{{
> 1) Tests_Term_getTerms::test_wp_delete_term_should_invalidate_cache
> Failed asserting that 1 matches expected 2.
> /var/www/tests/phpunit/tests/term/getTerms.php:181
> }}}
> {{{
> 1) Tests_Query_VerbosePageRules::test_page_page_2
> is_404 is true but is expected to be false.
> is_page is false but is expected to be true.
> is_paged is false but is expected to be true.
> is_singular is false but is expected to be true.
> /var/www/tests/phpunit/includes/abstract-testcase.php:971
> /var/www/tests/phpunit/tests/query/conditionals.php:327
>
> 1) Tests_Query_MetaQuery::test_meta_query_decimal_results
> Failed asserting that two arrays are equal.
> --- Expected
> +++ Actual
> @@ @@
>  Array (
> -    0 => 2414
>  )
> /var/www/tests/phpunit/includes/abstract-testcase.php:658
> /var/www/tests/phpunit/tests/query/metaQuery.php:1170
> }}}
> {{{
> 1) Tests_Term_getTerms::test_hierarchical_true_with_parent
> Failed asserting that two arrays are equal.
> --- Expected
> +++ Actual
> @@ @@
>  Array (
> -    0 => 1277
> -    1 => 1280
> +    0 => 1280
>  )
> /var/www/tests/phpunit/includes/abstract-testcase.php:658
> /var/www/tests/phpunit/tests/term/getTerms.php:2257
> }}}
> {{{
> 1) Tests_Query_SetupPostdata::test_secondary_query_nextpage
> Failed asserting that two arrays are identical.
> --- Expected
> +++ Actual
> @@ @@
>  Array &0 (
> -    0 => 'Post 1 Page 1'
> -    1 => 'Post 1 Page 2'
> +    0 => 'Post 2 Page 1'
> +    1 => 'Post 2 Page 2'
>  )
> /var/www/tests/phpunit/tests/query/setupPostdata.php:249
> }}}
> {{{
> 1) Tests_Query_MetaQuery::test_meta_between_not_between
> Failed asserting that 1 matches expected 2.
> /var/www/tests/phpunit/tests/query/metaQuery.php:1537
> }}}
> {{{
> 1)
> Tests_Post_getPages::test_get_pages_non_hierarchical_non_empty_child_of
> Failed asserting that two arrays are equal.
> --- Expected
> +++ Actual
> @@ @@
>  Array (
>      0 => 1677
>      1 => 1678
> -    2 => 1679
>  )
> /var/www/tests/phpunit/includes/abstract-testcase.php:649
> /var/www/tests/phpunit/tests/post/getPages.php:617
> }}}
> {{{
> 1) Tests_Post_Query::test_orderby_post__in_array
> Failed asserting that two arrays are identical.
> --- Expected
> +++ Actual
> @@ @@
>  Array &0 (
> -    0 => 428
> -    1 => 426
> -    2 => 429
> +    0 => 426
> +    1 => 429
>  )
> /var/www/tests/phpunit/tests/post/query.php:192
> }}}
> {{{
> 1) Tests_Get_Archives::test_wp_get_archives_limit
> Failed asserting that two strings are equal.
> --- Expected
> +++ Actual
> @@ @@
> -'<li><a href='http://example.org/?p=1143'>Post title 3347</a></li>\n
> -       <li><a href='http://example.org/?p=1142'>Post title
> 3346</a></li>\n
> +'<li><a href='http://example.org/?p=1142'>Post title 3346</a></li>\n
>         <li><a href='http://example.org/?p=1141'>Post title
> 3345</a></li>\n
>         <li><a href='http://example.org/?p=1140'>Post title
> 3344</a></li>\n
> -       <li><a href='http://example.org/?p=1139'>Post title
> 3343</a></li>'
> +       <li><a href='http://example.org/?p=1139'>Post title
> 3343</a></li>\n
> +       <li><a href='http://example.org/?p=1138'>Post title
> 3342</a></li>'
> /var/www/tests/phpunit/tests/functions/getArchives.php:79
>
> 2) Tests_Get_Archives::test_wp_get_archives_show_post_count
> Failed asserting that two strings are equal.
> --- Expected
> +++ Actual
> @@ @@
> -'<li><a href='http://example.org/?m=201909'>September
> 2019</a> (8)</li>'
> +'<li><a href='http://example.org/?m=201909'>September
> 2019</a> (7)</li>'
> /var/www/tests/phpunit/tests/functions/getArchives.php:127
> }}}
> {{{
> 1) Tests_Query_MetaQuery::test_meta_query_compare_not_exists
> Failed asserting that two arrays are equal.
> --- Expected
> +++ Actual
> @@ @@
>  Array (
>      0 => 990
>      1 => 991
> -    2 => 992
>  )
> /var/www/tests/phpunit/includes/abstract-testcase.php:649
> /var/www/tests/phpunit/tests/query/metaQuery.php:654
> }}}
> {{{
> 1)
> Test_WP_Customize_Nav_Menus::test_load_available_items_query_returns_post_item_with_page_number
> Failed asserting that an array contains Array &0 (
>     'id' => 'post-1030'
>     'title' => 'Post Title'
>     'type' => 'post_type'
>     'type_label' => 'Post'
>     'object' => 'post'
>     'object_id' => 1030
>     'url' => 'http://example.org/?p=1030'
> ).
> /var/www/tests/phpunit/tests/customize/nav-menus.php:182
> }}}
> {{{
> 1)
> Tests_Comment_Template::test_get_comments_number_text_declension_with_custom_args
> with data set #10 (2, '% Comments', '2 Comments')
> Failed asserting that two strings are equal.
> --- Expected
> +++ Actual
> @@ @@
> -'2 Comments'
> +'No Comments'
> /var/www/tests/phpunit/tests/comment/template.php:68
> }}}
> {{{
> 1) Tests_oEmbed_Discovery::test_add_oembed_discovery_links_to_attachment
> is_404 is true but is expected to be false.
> is_attachment is false but is expected to be true.
> is_single is false but is expected to be true.
> is_singular is false but is expected to be true.
> /var/www/tests/phpunit/includes/abstract-testcase.php:964
> /var/www/tests/phpunit/tests/oembed/discovery.php:81
> }}}
> {{{
> 1) Tests_Rewrite::test_url_to_postid_hierarchical_with_matching_leaves
> Failed asserting that 0 matches expected 3429.
> /var/www/tests/phpunit/tests/rewrite.php:245
> }}}
> {{{
> 1) Tests_Post_getPages::test_get_pages_meta
> Failed asserting that 0 matches expected 1.
> /var/www/tests/phpunit/tests/post/getPages.php:267
> }}}
> {{{
> 1)
> Tests_Link_GetNextCommentsLink::test_page_should_respect_value_of_cpage_query_var
> PHPUnit\Framework\Exception: Argument #2 (No Value) of
> PHPUnit\Framework\Assert::assertContains() must be a array, traversable
> or string
> /var/www/tests/phpunit/tests/link/getNextCommentsLink.php:19
> }}}
> {{{
> 1) Tests_Term_WpSetObjectTerms::test_set_object_terms_by_id
> Failed asserting that 4 matches expected 5.
> /var/www/tests/phpunit/tests/term/wpSetObjectTerms.php:128
> 2) Tests_Term_WpSetObjectTerms::test_set_object_terms_by_name
> Failed asserting that 4 matches expected 5.
> /var/www/tests/phpunit/tests/term/wpSetObjectTerms.php:161
> }}}
> {{{
> 1)
> Tests_Comment_Submission::test_submitting_comment_with_content_too_long_returns_error
> Failed asserting that two strings are identical.
> --- Expected
> +++ Actual
> @@ @@
> -'comment_content_column_length'
> +'comment_on_draft'
> /var/www/tests/phpunit/tests/comment-submission.php:687
> }}}
> {{{
> 1) Test_WP_Customize_Nav_Menu_Item_Setting::test_save_updated
> Failed asserting that an array is not empty.
> /var/www/tests/phpunit/tests/customize/nav-menu-item-setting.php:658
> }}}
> {{{
> 1) Tests_Query_Conditionals::test_is_single_with_attachment
> Failed asserting that false is true.
> /var/www/tests/phpunit/tests/query/conditionals.php:1025
> }}}
> {{{
> 1)
> Tests_Query_MetaQuery::test_meta_query_compare_exists_with_value_should_convert_to_equals
> Failed asserting that two arrays are equal.
> --- Expected
> +++ Actual
> @@ @@
>  Array (
> -    0 => 985
>  )
> /var/www/tests/phpunit/includes/abstract-testcase.php:684
> /var/www/tests/phpunit/tests/query/metaQuery.php:593
> }}}
> {{{
> 1) WP_Test_REST_Posts_Controller::test_get_items_author_query
> Failed asserting that 31 matches expected 32.
> /var/www/tests/phpunit/tests/rest-api/rest-posts-controller.php:269
> 2) WP_Test_REST_Posts_Controller::test_get_items_author_exclude_query
> Failed asserting that 31 matches expected 32.
> /var/www/tests/phpunit/tests/rest-api/rest-posts-controller.php:301
> 3) WP_Test_REST_Posts_Controller::test_get_items_search_query
> Failed asserting that 30 matches expected 31.
> /var/www/tests/phpunit/tests/rest-api/rest-posts-controller.php:510
> ...
> }}}
> {{{
> 1) Tests_Post::test_wp_count_posts_filtered
> Failed asserting that '2' matches expected 3.
> /var/www/tests/phpunit/tests/post.php:836
> }}}
> {{{
> 1)
> Tests_Rewrite_OldSlugRedirect::test_old_slug_doesnt_redirect_when_reused
> Failed asserting that two strings are equal.
> --- Expected
> +++ Actual
> @@ @@
> -'http://example.org/?p=3483/'
> +'http://example.org/foo-bar/'
> /var/www/tests/phpunit/tests/rewrite/oldSlugRedirect.php:144
> }}}

New description:

 Every few commits, there's now a random PHP 7.1 test failure on Travis,
 mostly off-by-one errors unrelated to the preceding commit. Started about
 1 month ago, after August 20th.

 Seems like either PHP 7.1 tests are for some reason more prone to random
 errors than other versions, or it's just more prominent now that random
 date/time failures from #45821 are fixed.

 Creating a ticket just for reference, any ideas as to why this happens
 appreciated.

 Some examples:
 {{{
 1) WP_Test_REST_Posts_Controller::test_get_items_search_query
 Failed asserting that 6 matches expected 7.
 /var/www/tests/phpunit/tests/rest-api/rest-posts-controller.php:459
 }}}
 {{{
 1) WP_Test_REST_Posts_Controller::test_get_items_pagination_headers
 Failed asserting that 49 matches expected 50.
 /var/www/tests/phpunit/tests/rest-api/rest-posts-controller.php:1157
 }}}
 {{{
 1) Tests_Canonical_PageOnFront::test with data set #3 ('/?page=1', '/')
 Failed asserting that two strings are equal.
 --- Expected
 +++ Actual
 @@ @@
 -'/'
 +'/?page=1'
 /var/www/tests/phpunit/includes/testcase-canonical.php:268
 /var/www/tests/phpunit/tests/canonical/pageOnFront.php:39

 2)
 Tests_Query_MetaQuery::test_meta_query_single_query_compare_arithmetic_comparisons
 Failed asserting that two arrays are equal.
 --- Expected
 +++ Actual
 @@ @@
  Array (
 -    0 => 2319
  )
 /var/www/tests/phpunit/includes/abstract-testcase.php:658
 /var/www/tests/phpunit/tests/query/metaQuery.php:162
 }}}
 {{{
 1) Tests_Query::test_tag_querystring_single_term
 Failed asserting that two arrays are equal.
 --- Expected
 +++ Actual
 @@ @@
  Array (
      0 => 2022
 -    1 => 2023
  )
 /var/www/tests/phpunit/includes/abstract-testcase.php:658
 /var/www/tests/phpunit/tests/query.php:321
 }}}
 {{{
 1) Tests_Term_getTerms::test_wp_delete_term_should_invalidate_cache
 Failed asserting that 1 matches expected 2.
 /var/www/tests/phpunit/tests/term/getTerms.php:181
 }}}
 {{{
 1) Tests_Query_VerbosePageRules::test_page_page_2
 is_404 is true but is expected to be false.
 is_page is false but is expected to be true.
 is_paged is false but is expected to be true.
 is_singular is false but is expected to be true.
 /var/www/tests/phpunit/includes/abstract-testcase.php:971
 /var/www/tests/phpunit/tests/query/conditionals.php:327

 1) Tests_Query_MetaQuery::test_meta_query_decimal_results
 Failed asserting that two arrays are equal.
 --- Expected
 +++ Actual
 @@ @@
  Array (
 -    0 => 2414
  )
 /var/www/tests/phpunit/includes/abstract-testcase.php:658
 /var/www/tests/phpunit/tests/query/metaQuery.php:1170
 }}}
 {{{
 1) Tests_Term_getTerms::test_hierarchical_true_with_parent
 Failed asserting that two arrays are equal.
 --- Expected
 +++ Actual
 @@ @@
  Array (
 -    0 => 1277
 -    1 => 1280
 +    0 => 1280
  )
 /var/www/tests/phpunit/includes/abstract-testcase.php:658
 /var/www/tests/phpunit/tests/term/getTerms.php:2257
 }}}
 {{{
 1) Tests_Query_SetupPostdata::test_secondary_query_nextpage
 Failed asserting that two arrays are identical.
 --- Expected
 +++ Actual
 @@ @@
  Array &0 (
 -    0 => 'Post 1 Page 1'
 -    1 => 'Post 1 Page 2'
 +    0 => 'Post 2 Page 1'
 +    1 => 'Post 2 Page 2'
  )
 /var/www/tests/phpunit/tests/query/setupPostdata.php:249
 }}}
 {{{
 1) Tests_Query_MetaQuery::test_meta_between_not_between
 Failed asserting that 1 matches expected 2.
 /var/www/tests/phpunit/tests/query/metaQuery.php:1537
 }}}
 {{{
 1) Tests_Post_getPages::test_get_pages_non_hierarchical_non_empty_child_of
 Failed asserting that two arrays are equal.
 --- Expected
 +++ Actual
 @@ @@
  Array (
      0 => 1677
      1 => 1678
 -    2 => 1679
  )
 /var/www/tests/phpunit/includes/abstract-testcase.php:649
 /var/www/tests/phpunit/tests/post/getPages.php:617
 }}}
 {{{
 1) Tests_Post_Query::test_orderby_post__in_array
 Failed asserting that two arrays are identical.
 --- Expected
 +++ Actual
 @@ @@
  Array &0 (
 -    0 => 428
 -    1 => 426
 -    2 => 429
 +    0 => 426
 +    1 => 429
  )
 /var/www/tests/phpunit/tests/post/query.php:192
 }}}
 {{{
 1) Tests_Get_Archives::test_wp_get_archives_limit
 Failed asserting that two strings are equal.
 --- Expected
 +++ Actual
 @@ @@
 -'<li><a href='http://example.org/?p=1143'>Post title 3347</a></li>\n
 -       <li><a href='http://example.org/?p=1142'>Post title
 3346</a></li>\n
 +'<li><a href='http://example.org/?p=1142'>Post title 3346</a></li>\n
         <li><a href='http://example.org/?p=1141'>Post title
 3345</a></li>\n
         <li><a href='http://example.org/?p=1140'>Post title
 3344</a></li>\n
 -       <li><a href='http://example.org/?p=1139'>Post title 3343</a></li>'
 +       <li><a href='http://example.org/?p=1139'>Post title
 3343</a></li>\n
 +       <li><a href='http://example.org/?p=1138'>Post title 3342</a></li>'
 /var/www/tests/phpunit/tests/functions/getArchives.php:79

 2) Tests_Get_Archives::test_wp_get_archives_show_post_count
 Failed asserting that two strings are equal.
 --- Expected
 +++ Actual
 @@ @@
 -'<li><a href='http://example.org/?m=201909'>September
 2019</a> (8)</li>'
 +'<li><a href='http://example.org/?m=201909'>September
 2019</a> (7)</li>'
 /var/www/tests/phpunit/tests/functions/getArchives.php:127
 }}}
 {{{
 1) Tests_Query_MetaQuery::test_meta_query_compare_not_exists
 Failed asserting that two arrays are equal.
 --- Expected
 +++ Actual
 @@ @@
  Array (
      0 => 990
      1 => 991
 -    2 => 992
  )
 /var/www/tests/phpunit/includes/abstract-testcase.php:649
 /var/www/tests/phpunit/tests/query/metaQuery.php:654
 }}}
 {{{
 1)
 Test_WP_Customize_Nav_Menus::test_load_available_items_query_returns_post_item_with_page_number
 Failed asserting that an array contains Array &0 (
     'id' => 'post-1030'
     'title' => 'Post Title'
     'type' => 'post_type'
     'type_label' => 'Post'
     'object' => 'post'
     'object_id' => 1030
     'url' => 'http://example.org/?p=1030'
 ).
 /var/www/tests/phpunit/tests/customize/nav-menus.php:182
 }}}
 {{{
 1)
 Tests_Comment_Template::test_get_comments_number_text_declension_with_custom_args
 with data set #10 (2, '% Comments', '2 Comments')
 Failed asserting that two strings are equal.
 --- Expected
 +++ Actual
 @@ @@
 -'2 Comments'
 +'No Comments'
 /var/www/tests/phpunit/tests/comment/template.php:68
 }}}
 {{{
 1) Tests_oEmbed_Discovery::test_add_oembed_discovery_links_to_attachment
 is_404 is true but is expected to be false.
 is_attachment is false but is expected to be true.
 is_single is false but is expected to be true.
 is_singular is false but is expected to be true.
 /var/www/tests/phpunit/includes/abstract-testcase.php:964
 /var/www/tests/phpunit/tests/oembed/discovery.php:81
 }}}
 {{{
 1) Tests_Rewrite::test_url_to_postid_hierarchical_with_matching_leaves
 Failed asserting that 0 matches expected 3429.
 /var/www/tests/phpunit/tests/rewrite.php:245
 }}}
 {{{
 1) Tests_Post_getPages::test_get_pages_meta
 Failed asserting that 0 matches expected 1.
 /var/www/tests/phpunit/tests/post/getPages.php:267
 }}}
 {{{
 1)
 Tests_Link_GetNextCommentsLink::test_page_should_respect_value_of_cpage_query_var
 PHPUnit\Framework\Exception: Argument #2 (No Value) of
 PHPUnit\Framework\Assert::assertContains() must be a array, traversable or
 string
 /var/www/tests/phpunit/tests/link/getNextCommentsLink.php:19
 }}}
 {{{
 1) Tests_Term_WpSetObjectTerms::test_set_object_terms_by_id
 Failed asserting that 4 matches expected 5.
 /var/www/tests/phpunit/tests/term/wpSetObjectTerms.php:128
 2) Tests_Term_WpSetObjectTerms::test_set_object_terms_by_name
 Failed asserting that 4 matches expected 5.
 /var/www/tests/phpunit/tests/term/wpSetObjectTerms.php:161
 }}}
 {{{
 1)
 Tests_Comment_Submission::test_submitting_comment_with_content_too_long_returns_error
 Failed asserting that two strings are identical.
 --- Expected
 +++ Actual
 @@ @@
 -'comment_content_column_length'
 +'comment_on_draft'
 /var/www/tests/phpunit/tests/comment-submission.php:687
 }}}
 {{{
 1) Test_WP_Customize_Nav_Menu_Item_Setting::test_save_updated
 Failed asserting that an array is not empty.
 /var/www/tests/phpunit/tests/customize/nav-menu-item-setting.php:658
 }}}
 {{{
 1) Tests_Query_Conditionals::test_is_single_with_attachment
 Failed asserting that false is true.
 /var/www/tests/phpunit/tests/query/conditionals.php:1025
 }}}
 {{{
 1)
 Tests_Query_MetaQuery::test_meta_query_compare_exists_with_value_should_convert_to_equals
 Failed asserting that two arrays are equal.
 --- Expected
 +++ Actual
 @@ @@
  Array (
 -    0 => 985
  )
 /var/www/tests/phpunit/includes/abstract-testcase.php:684
 /var/www/tests/phpunit/tests/query/metaQuery.php:593
 }}}
 {{{
 1) WP_Test_REST_Posts_Controller::test_get_items_author_query
 Failed asserting that 31 matches expected 32.
 /var/www/tests/phpunit/tests/rest-api/rest-posts-controller.php:269
 2) WP_Test_REST_Posts_Controller::test_get_items_author_exclude_query
 Failed asserting that 31 matches expected 32.
 /var/www/tests/phpunit/tests/rest-api/rest-posts-controller.php:301
 3) WP_Test_REST_Posts_Controller::test_get_items_search_query
 Failed asserting that 30 matches expected 31.
 /var/www/tests/phpunit/tests/rest-api/rest-posts-controller.php:510
 ...
 }}}
 {{{
 1) Tests_Post::test_wp_count_posts_filtered
 Failed asserting that '2' matches expected 3.
 /var/www/tests/phpunit/tests/post.php:836
 }}}
 {{{
 1)
 Tests_Rewrite_OldSlugRedirect::test_old_slug_doesnt_redirect_when_reused
 Failed asserting that two strings are equal.
 --- Expected
 +++ Actual
 @@ @@
 -'http://example.org/?p=3483/'
 +'http://example.org/foo-bar/'
 /var/www/tests/phpunit/tests/rewrite/oldSlugRedirect.php:144
 }}}
 {{{
 1)
 Tests_Rewrite_AddRewriteEndpoint::test_permalink_endpoint_only_applies_on_permalink
 Failed asserting that false is true.
 /var/www/tests/phpunit/tests/rewrite/addRewriteEndpoint.php:88
 }}}
 {{{
 1) Tests_Post::test_wp_count_posts
 Failed asserting that 0 matches expected 1.
 /var/www/tests/phpunit/tests/post.php:820
 }}}

--

-- 
Ticket URL: <https://core.trac.wordpress.org/ticket/48145#comment:24>
WordPress Trac <https://core.trac.wordpress.org/>
WordPress publishing platform


More information about the wp-trac mailing list