[wp-trac] [WordPress Trac] #60805: Reading Settings: add option to discourage AI services from crawling the site

[WordPress Trac]

#60805: Reading Settings: add option to discourage AI services from crawling the
site
-----------------------------+------------------------------
 Reporter:  jeherve          |       Owner:  (none)
     Type:  feature request  |      Status:  new
 Priority:  normal           |   Milestone:  Awaiting Review
Component:  Privacy          |     Version:
 Severity:  normal           |  Resolution:
 Keywords:                   |     Focuses:  privacy
-----------------------------+------------------------------

Comment (by peterwilsoncc):

 My inclination for now is that this is best left for plugins for the time
 being.

 * AI companies and crawlers are increasing at a rapid rate, so it's
 unlikely Core contributors would be able to keep up with the known bots.
 That puts WP in a position in which it is, arguably, favoring some
 companies over others.
 * It's pretty well known that some AI crawlers respect robots.txt, while
 others ignore it. The most effective way of blocking them is via their IP
 address. This isn't something that can be done in Core due to full page
 caching, the use of CDNs. In effect, robots.txt in core would affect the
 more ethical providers.

 On my personal site, I'd probably install such a plugin but that doesn't
 necessarily mean it's appropriate for core.

-- 
Ticket URL: <https://core.trac.wordpress.org/ticket/60805#comment:9>
WordPress Trac <https://core.trac.wordpress.org/>
WordPress publishing platform