[wp-meta] [Making WordPress.org] #1181: Enhancement for Reviews/Ratings: Use a Rolling Time Frame for more current reviews

[Making WordPress.org]

#1181: Enhancement for Reviews/Ratings: Use a Rolling Time Frame for more current
reviews
----------------------------+---------------------
 Reporter:  cais            |       Owner:  (none)
     Type:  enhancement     |      Status:  new
 Priority:  low             |   Milestone:  Q1
Component:  Support Forums  |  Resolution:
 Keywords:                  |
----------------------------+---------------------

Comment (by dd32):

 This is mostly me thinking out-loud after reviewing the raw data that
 powers the rating number. I apologise for the length and the random data
 points without any specific examples or data backing it up.

 I pulled the Plugin Review data for all-time vs "Ratings from 6mths prior
 to the last update" and went from there.
 Insights:
  - Most plugins have next-to-no reviews, and limiting it to the last 6
 months just causes a single 5 star (or 1 star) review to be the number
 behind it
  - Some plugins have a lot of reviews, for example the top rated plugin
 (Which prompts users who actively use it to review IIRC) has 5x the
 reviews of the 2nd most reviewed plugin.
  - Most large plugins (>1m installs) who have a decent stream of reviews
 coming in, generally have the same rating for all-time vs the 6mth time
 frame. Maybe it's 0.1 stars off, but it's rare for it to differ much.

 In the data I've been working with
  - 198 plugins have >10 reviews in six months and would have their rating
 increased by at least 0.1 stars (only 19 would gain 0.5 stars). If you
 limit it to >20 reviews, then it's 95 gaining 0.1, and 7 gaining more than
 0.5.
  - 330 plugins with >10 reviews in six months would have their rating
 decreased by more than 0.1 stars (93 loosing more than 0.5 stars, and 29
 loosing more than 1 star). Once again, >20 reviews is 112 decreasing more
 than 0.1, 32 more than 0.5, and 15 loosing more than 1 star.
  - 7,500 plugins would see less than a 0.1 change in their review, only
 593 of those have >10 reviews
  - A LOT of plugins who have never been reviewed, or have not been
 reviewed in the 6 months prior to their last update.

 So.. options IMHO:
  - Use recent reviews if it theres `>20` reviews. Ie. `$reviews =
 $six_month_num_reviews >= 20 ? $six_month_rating : $all_time_rating;` but
 that opens up more gaming, where you can 'fix' a bad rating temporarily by
 spamming or drop a plugin rating by spamming (something which is seen more
 often than I like)
  - Weighted ratings, more recent reviews are weighed heavier than old
 reviews
   - In my small amount of experimentation with that, it seemed to mostly
 match up with the averages of current with a significant impact upon
 performance of measuring the data
  - Weighted ratings by time period, `$rating = avg( $six_mth_rating,
 $all_time_rating )` or `$rating = avg( $this_month_rating,
 $last_six_month_rating, $all_time_rating )`
  - Leave it as-is, all-time reviews. Revisit when more plugins are getting
 reviewed and there's more than 10 points of data for the majority of
 plugins.

 I'm leaning towards no change here, purely because of the significant
 limited data points available for most plugins and the majority of plugins
 having no change.
 The 15..29 plugins who would loose more than 1 star is a good indication
 that we need to start to consider changing something though, but I don't
 think a hard 6mth cut off (either of the last 6mths, or 6mths prior to
 last update) makes the rating number more accurate for most end-users.
 Using any of the weighted averages or multiple range averages also tends
 to hide these plugins as the rating drop is more like 0.3~0.5 star drop at
 most after applying that averaging.

-- 
Ticket URL: <https://meta.trac.wordpress.org/ticket/1181#comment:25>
Making WordPress.org <https://meta.trac.wordpress.org/>
Making WordPress.org