[wp-meta] [Making WordPress.org] #4068: Change 503 behaviour on paginated states

[Making WordPress.org]

#4068: Change 503 behaviour on paginated states
----------------------------+-----------------------
 Reporter:  jonoaldersonwp  |       Owner:  dd32
     Type:  defect          |      Status:  accepted
 Priority:  high            |   Milestone:
Component:  General         |  Resolution:
 Keywords:  seo             |
----------------------------+-----------------------

Comment (by dd32):

 Just noting that there were two different bot blocks in place
  - A requests per second limit = 503 (now a 429 and increased limits)
  - A pagination limit for >= `/page/50` = 404

 I originally missed the first block which this ticket was actually about,
 and thought it was the second. Limiting the paginated results to ~50
 "solves" the latter point, and also reduces the number of URLs to be
 crawled which should help the first block (Less requests overall).

 There's also a few issues around the high-pagination block as implemented,
 as Google IS indexing high-paginated URLs (..under multiple variations due
 to junk args appended to the url..) however it doesn't seem like something
 we want to encourage given the requests/second block is being triggered.

-- 
Ticket URL: <https://meta.trac.wordpress.org/ticket/4068#comment:13>
Making WordPress.org <https://meta.trac.wordpress.org/>
Making WordPress.org