[wp-trac] [WordPress Trac] #33156: Allow admin-ajax crawling

[WordPress Trac]

#33156: Allow admin-ajax crawling
-----------------------------------+-----------------------------
 Reporter:  joostdevalk            |       Owner:  SergeyBiryukov
     Type:  enhancement            |      Status:  closed
 Priority:  normal                 |   Milestone:  4.4
Component:  General                |     Version:
 Severity:  normal                 |  Resolution:  fixed
 Keywords:  2nd-opinion has-patch  |     Focuses:
-----------------------------------+-----------------------------

Comment (by dmchale):

 Replying to [comment:22 KnowingArt_com]:
 > I'm told the WordPress philosophy is to make decisions instead of
 offering options. You don't make certain decisions without asking my
 permission.

 In fairness, that's exactly what it means to make a decision and not offer
 an option.

 That said, this ticket was discussing the default behavior of the
 robots.txt file. You can short-circuit what WordPress does out of the box
 in one of two ways.

 1. Create a physical robots.txt file on your server. If WordPress detects
 a physical file at the web root, it will not add to / remove from / modify
 that file in any way (this includes any plugins that dynamically modify
 the robots.txt file as well)

 2. Use the `robots_txt` filter to modify the contents of the WordPress
 defaults https://developer.wordpress.org/reference/hooks/robots_txt/

-- 
Ticket URL: <https://core.trac.wordpress.org/ticket/33156#comment:24>
WordPress Trac <https://core.trac.wordpress.org/>
WordPress publishing platform