[wp-trac] [WordPress Trac] #57687: Add automated performance testing CI workflow MVP

[WordPress Trac]

#57687: Add automated performance testing CI workflow MVP
--------------------------------------+--------------------------
 Reporter:  adamsilverstein           |       Owner:  mukesh27
     Type:  task (blessed)            |      Status:  assigned
 Priority:  normal                    |   Milestone:  6.2
Component:  Build/Test Tools          |     Version:  trunk
 Severity:  normal                    |  Resolution:
 Keywords:  has-patch has-unit-tests  |     Focuses:  performance
--------------------------------------+--------------------------

Comment (by joemcgill):

 Thanks @desrosj, @flixos90, @costdev, @youknowriad, and @swissspidy for
 providing feedback on the PR. I believe that all of the feedback has been
 addressed and am looking to finalize and commit this new workflow before
 the 6.2 branch is created.

 One important aspect of this workflow that I think is worthy of future
 discussion, but I don't see as a blocker to this being committed, is the
 strategy being used to normalize each measurement against a baseline
 version in order to reduce the variance that can be introduced by the
 GitHub runners themselves.

 The basic idea here is that each time we take a set of measurements, we
 are also measuring the performance of a consistent version of WP that we
 can use as a baseline. Because that baseline version stays consistent, we
 can attribute differences in measurements of the baseline version to the
 environment, and not to differences in the code and remove that variance
 from the measurement we're taking on each commit in the reports that are
 generated in the https://codevitals.run dashboard. It should also be noted
 that none of that normalization happens as part of this workflow, we're
 just providing data to the reporting tool, which handles those
 calculations.

 For now, we're using the most recent release (WP 6.1.1) as the baseline.
 The Gutenberg project uses the same strategy for their metrics, but
 updates the base version each time a new major version of WP is released.
 I don't think we will want to follow the same practice for core and will
 instead maintaining the same baseline over longer periods of time, but
 that's mainly a question for future consideration once we've had some time
 to monitor these metrics.

 I expect us to iterate over our collection and reporting processes over
 time, but this current workflow should give us a good starting foundation
 from which to learn and improve.

-- 
Ticket URL: <https://core.trac.wordpress.org/ticket/57687#comment:12>
WordPress Trac <https://core.trac.wordpress.org/>
WordPress publishing platform