[wp-meta] [Making WordPress.org] #3192: Provide an API endpoint to fetch plugin checksums

[Making WordPress.org]

#3192: Provide an API endpoint to fetch plugin checksums
------------------------------+-----------------------
 Reporter:  schlessera        |       Owner:  dd32
     Type:  enhancement       |      Status:  accepted
 Priority:  normal            |   Milestone:
Component:  Plugin Directory  |  Resolution:
 Keywords:                    |
------------------------------+-----------------------
Changes (by dd32):

 * owner:   => dd32
 * status:  new => accepted


Comment:

 Okay, so, there's a few options for generating checksums, I'm going to
 outline to two main ones which I'd investigated briefly, and ended up
 writing a POC for both of:

 ** Create checksums when we create ZIPs **
 ie. we create `plugin-slug.1.2.3.zip` and also create `plugin-
 slug.1.2.3.checksums.json`

 This seems like the simplest option, but it's also the most unreliable and
 useless method available.
  - 49% of WordPress.org plugin packages have a version in the filename,
 but in ~8% of those it's not actually the same as what version of plugin
 is inside the ZIP.
    - The reason the versions are mismatched, is svn tag vs plugin header
 not matching.
  - 51% of Zips don't include a version at all (being built from trunk).

 That means only 45% in the end has a Zip filename == plugin version match.

 ** Create checksums when we create ZIPs, but based on the plugin header
 version **

 This seems to be the best option to me. Plugin Zips should've been done
 like this too IMHO.

 When generating zips, we create a json file of the checksums. If the file
 exists of that version already, we update it to include the previous, and
 current checksums.
 For example:
  - create `example-plugin/tags/v2.0` with `Version: 2.0.1` we create
 `example-plugin.v2.0.zip` already and we'll add `example-
 plugin.2.0.1.checksums.json`.
  - update `example-plugin/trunk/plugin.php` as stable_tag at version 2.0.1
 with a new version and we'll create `example-plugin.zip` and `example-
 plugin.2.0.1.checksums.json`.

 We wouldn't create checksums for the development version of a plugin (ie.
 `/trunk/`).
 We would create checksums if `/trunk/` is the `stable_tag` though,
 following the above file naming.

 Creating checksum files for older versions, where it's being released from
 `/trunk/` or tag modifications have been made is unlikely, but we'd have
 them going forward.

 ** Database table of checksums **
 Another option raised was storing it in a DB table instead of flat files.
 This doesn't seem sane to me from a caching / updating perspective, the
 amount of data would increase exponentially, and not actually use any
 features of SQL other than a key-value store.
 (Quick maths: 50k plugins * 15 versions on average * 10 files on average =
 7.5million rows, that's not taking into consideration that larger plugins
 have more files and would skew it upwards IMHO)

 ** Themes **

 Themes are strangely much easier.
  - Can't have multiple versions of a file in any given version.
  - ZIP file names contain the Version Header
  - SVN tags are based on the Version Header
 Theme Authors don't have access to SVN, so all the version numbers are
 handled correctly without human error.

 Checksums could therefor be generated/served very easily there, but we'd
 probably leave that until the ZIP storage method for themes is updated to
 a similar process that we use for Plugins.

--
Ticket URL: <https://meta.trac.wordpress.org/ticket/3192#comment:2>
Making WordPress.org <https://meta.trac.wordpress.org/>
Making WordPress.org