[wp-trac] [WordPress Trac] #43557: Remove wp-admin/admin-ajax.php from robots.txt - sure give ppl the ability to add it, remove the default

[WordPress Trac]

#43557: Remove wp-admin/admin-ajax.php from robots.txt - sure give ppl the ability
to add it, remove the default
-------------------------------+-----------------------------
 Reporter:  phuquocgreentours  |      Owner:
     Type:  defect (bug)       |     Status:  new
 Priority:  normal             |  Milestone:  Awaiting Review
Component:  General            |    Version:  4.9.4
 Severity:  normal             |   Keywords:
  Focuses:                     |
-------------------------------+-----------------------------
 This is a follow-up to #33156.

 This is dumb...

 My site returns 0 and HTTP status code of 400 when hitting admin-ajax.php.
 Of course, Google is going to have a hard time interpreting that.

 All the links on the page Google complained about are Javascript executed
 requests. There are no bare https://site/wp-admin/admin-
 ajax.php?someparam=getsomedata that Googlebot can follow. Everything is
 executed from JS and obviously requires the correct params.

 I don't want errors from Googlebot...

 Looking for how to remove this default robots.txt rubbish.

 Expose robots.txt to the WP API so whoever needs this can add it.
 Making it default is ludicrous :/

--
Ticket URL: <https://core.trac.wordpress.org/ticket/43557>
WordPress Trac <https://core.trac.wordpress.org/>
WordPress publishing platform