[wp-trac] [WordPress Trac] #60227: HTML API: Add external test suite

[WordPress Trac]

#60227: HTML API: Add external test suite
--------------------------------------+---------------------
 Reporter:  jonsurrell                |       Owner:  (none)
     Type:  enhancement               |      Status:  new
 Priority:  normal                    |   Milestone:  6.5
Component:  Build/Test Tools          |     Version:
 Severity:  normal                    |  Resolution:
 Keywords:  has-patch has-unit-tests  |     Focuses:
--------------------------------------+---------------------

Comment (by jonsurrell):

 I'll try to justify the skipped tests and address the performance
 concerns.

 [comment:23 dmsnell already mentioned] that these tests are not expected
 to be skipped indefinitely. As more of the HTML API is completed, fewer
 tests should be skipped. I don't know whether we'll ever reach 0 skips,
 but if we do reach a point where we have skipped tests we never expect to
 pass, I'm sure we can find a mechanism to avoid running them.

 There are few reasons skipping makes sense at this time.

 We do want them to pass eventually.

 We can't avoid running the tests in most cases because the HTML API
 implementation informs the test that it should be skipped. If the HTML API
 does not support the encountered tags or detects what it determines to be
 an incomplete HTML fragment (e.g. it ends on an incomplete tag `<texta`),
 we don't expect the tests to pass. These aren't ''failures'', they're
 parts of the API that are incomplete. Because the test is running when we
 get this information from the HTML API, the test must have a status. We
 don't want to fail the entire test suite for known and expected
 limitations. We could let the tests pass, but we lose valuable information
 about the status of the HTML API overall.

 Given this is an external suite of tests, I don't think it makes sense to
 modify the source in any way.

 With all of that in mind, skipping (or marking incomplete) these tests
 seems like the best option to me.


 ----

 There were some performance concerns raised regarding the skipped tests.
 I'd like to provide some data. I've isolated the html5lib-tests suite and
 done some benchmarking. Over 30 runs, the entire test suite (including all
 the skipped tests) runs in under 750ms on an M2 MacBook Pro.

   Time (mean ± σ):     722.2 ms ±   8.6 ms    [User: 673.6 ms, System:
 38.2 ms]
   Range (min … max):   706.1 ms … 741.5 ms    30 runs

 I also compared the complete phpunit test suite on trunk and the html5lib-
 tests branch on the same machine. Trunk averaged ~202 seconds and the
 branch ~203 seconds.

-- 
Ticket URL: <https://core.trac.wordpress.org/ticket/60227#comment:26>
WordPress Trac <https://core.trac.wordpress.org/>
WordPress publishing platform