[wp-trac] [WordPress Trac] #33156: Allow admin-ajax crawling

[WordPress Trac]

#33156: Allow admin-ajax crawling
-----------------------------------+-----------------------------
 Reporter:  joostdevalk            |       Owner:  SergeyBiryukov
     Type:  enhancement            |      Status:  closed
 Priority:  normal                 |   Milestone:  4.4
Component:  General                |     Version:
 Severity:  normal                 |  Resolution:  fixed
 Keywords:  2nd-opinion has-patch  |     Focuses:
-----------------------------------+-----------------------------

Comment (by KnowingArt_com):

 It seems like all the comments of concern were ignored.

 WordPress does not need permission from robots.txt to access itself. Take
 a step back and ask yourself why robots are "allowed" access to wp-admin
 /admin-ajax.php. It makes no sense, as if people forgot the role of
 robots.txt

 Robots.txt does not exist to manage all conceivable non-human
 interactions. It was originally created to save bandwidth, because one gig
 of transfer could cost over $10. If you simply crawled somebody's website
 back in 1999, they might threaten to sue you. Ask me how I know. Frankly,
 bandwidth and CPU are cheap enough now that robots.txt should be obsolete.

 After some decades and bazillions of pageviews, I have never used a
 robots.txt, because I don't want to discourage robots from enjoying my
 content. I don't know the exact version of WordPress this changed, but it
 seems WordPress decided to make robots.txt mandatory. In my expert
 opinion, that was a foolish decision.

 Respectful bots will avoid /wp-admin/ without being told. Disrespectful
 bots will do whatever they want. This auto-generated robots.txt is
 unnecessary, it just creates confusion and solves nothing.

 I'm told the WordPress philosophy is to make decisions instead of offering
 options. You don't make certain decisions without asking my permission.
 I'm drawing the line at robots.txt, I see this as a violation where
 WordPress is claiming ownership of something that does not belong to
 WordPress. (My server, my choice.) If I "allow" a robot onto my server,
 that is my decision to make as a system admin. Just because the average
 WordPress user is not very sophisticated with technology, that doesn't
 mean you can just take control of whatever you want, just because I gave
 you permission to auto-install upgrades.

 What's next? Are you going to try creeping into php.ini? Seriously, this
 should be a concern as larger companies are allowed to submit code to
 WordPress. If you're going to draw a line somewhere, might as well draw
 the line with robots.txt

 As for admin-ajax.php, whoever added that line should at least include a
 robots.readme to explain why robots.txt is mandatory, why it makes no
 sense, the relationship to wp-sitemap.xml, and include a link back to this
 URL, because it took me two days to follow the breadcrumbs back to this
 ticket. To say the least, this is not how I wanted to spend my week. And
 after digging into pages and pages of explanations, I'm still wondering
 why anyone would add admin-ajax.php to robots.txt

 "Since it's often used on front-end."

 Like I said, the front-end has nothing to do with robots.txt

 "What might be the downside of allowing admin-ajax.php to be crawled? Any
 chance of unwanted content appearing in SERPs?"

 BINGO! I'm having a problem with DuckDuckGo right now, it's listing /wp-
 admin/ as the #1 search result for my domain.

-- 
Ticket URL: <https://core.trac.wordpress.org/ticket/33156#comment:22>
WordPress Trac <https://core.trac.wordpress.org/>
WordPress publishing platform