[wp-trac] [WordPress Trac] #33156: Allow admin-ajax crawling

[WordPress Trac]

#33156: Allow admin-ajax crawling
-----------------------------------+-----------------------------
 Reporter:  joostdevalk            |       Owner:  SergeyBiryukov
     Type:  enhancement            |      Status:  closed
 Priority:  normal                 |   Milestone:  4.4
Component:  General                |     Version:
 Severity:  normal                 |  Resolution:  fixed
 Keywords:  2nd-opinion has-patch  |     Focuses:
-----------------------------------+-----------------------------

Comment (by KnowingArt_com):

 Replying to [comment:22 KnowingArt_com]:
 > It seems like all the comments of concern were ignored.

 I'm starting to get a better understanding of the issue(s) that triggered
 this desire to sortof whitelist this ajax script for Googlebot. However,
 we should not pollute robots.txt to fix poorly-conceived AJAX themes that
 lazyload content without non-AJAX placeholder content, do not fail
 gracefully, or whatever happened to these themes.

 Likewise, robots.txt is not the best place for Googlebot-specific problems
 either. I think WordPress is big enough to work that out with Google
 directly. And for really specific problems, just use your web server
 config.

 I'm also thinking, if WordPress decides to back out of robots.txt, will
 that break stuff? I doubt it, but I don't know for sure. I think the
 alternative is worse. Because now you have wp-sitemap.xml in there, how
 deep will this rabbit hole go before all the custom robots.txt files out
 there the need to be rewritten to "catch up" with the auto-generated
 WordPress robots.txt?

 Also, I created this last night...

 https://wordpress.stackexchange.com/questions/403753/why-does-do-robots-
 allow-wp-admin-admin-ajax-php-by-default

-- 
Ticket URL: <https://core.trac.wordpress.org/ticket/33156#comment:25>
WordPress Trac <https://core.trac.wordpress.org/>
WordPress publishing platform