[wp-trac] [WordPress Trac] #60227: HTML API: Add external test suite

[WordPress Trac]

#60227: HTML API: Add external test suite
--------------------------------------+---------------------
 Reporter:  jonsurrell                |       Owner:  (none)
     Type:  enhancement               |      Status:  new
 Priority:  normal                    |   Milestone:  6.5
Component:  Build/Test Tools          |     Version:
 Severity:  normal                    |  Resolution:
 Keywords:  has-patch has-unit-tests  |     Focuses:
--------------------------------------+---------------------

Comment (by hellofromTonya):

 Replying to [comment:23 dmsnell]:
 >That being said, it sounds like the main complaint is people thinking
 that skipped tests implies they should be doing more work on something
 they are unfamiliar with?

 From me, no this isn't the case.

 My concerns are:

 * Impacts to the test suite performance.
 >these skips also consume resources such as adding processing time to the
 test suite.

 Efforts have been and are underway to make the test suite faster and use
 less resources. The more non-actionable tasks its does (such as skipping
 tests), the slower it runs and more the resources it uses.

 * Harder for those not working on the HTML API to parse the results.

 >the skipped tests messages pollute the test suite's results, making it
 harder for contributors (of other tests and code beyond the HTML API) to
 scan through the results to find relevant information.

 While I appreciate the intent and impact to those working on the HTML API,
 my concerns are on the impacts to Core's test suite and the CI jobs.

 Thus, I'm curious:
 What is possible to avoid skipping and not running unsupported datasets in
 the tests?

 >manually sift through the thousands of tests and break them up into
 pieces

 Hmm, this raises another concern:

 What is the impact to Core's test suite runtime and resource consumption?

 As I noted previously, an effort has been and is underway to improve the
 test suite's runtime and performance. While this effort should not block
 adding more and more code coverage, where possible, the performance of the
 test suite should also be considered.

 So if hundreds or thousands of tests are unsupported and will be skipped,
 IMO these should not be yielded as a test dataset, i.e. at least not
 without assessing the impacts to the test suite itself.

-- 
Ticket URL: <https://core.trac.wordpress.org/ticket/60227#comment:24>
WordPress Trac <https://core.trac.wordpress.org/>
WordPress publishing platform