[wp-trac] [WordPress Trac] #43590: Search Engine Visibility option does not work as intended

[WordPress Trac]

#43590: Search Engine Visibility option does not work as intended
--------------------------+------------------------------
 Reporter:  mamaedler     |       Owner:  (none)
     Type:  defect (bug)  |      Status:  new
 Priority:  normal        |   Milestone:  Awaiting Review
Component:  General       |     Version:  4.9.4
 Severity:  normal        |  Resolution:
 Keywords:  has-patch     |     Focuses:  administration
--------------------------+------------------------------

Comment (by mamaedler):

 @jonoaldersonwp Thanks for your insights.

 I have an idea. How about adding robots.txt back in and change the
 ruleset, so it does allow crawlers to read the "home page" with our
 "noindex meta tag" and disallow everything else, all common sub folders
 (including assets)?

 I think the best solution would be an appropriate X-Robots-Tag in the
 **HTTP response header**, like you already mentioned. Every resource
 (html, images, videos, even JavaScript and style sheets) would have the
 noindex directive in its http header if we applied this via Apache
 directives.

 Is it save to assume WordPress runs on Apache? Then this could be done via
 an `.htaccess` file that sits in the root directory and has the following
 content.

 {{{
 <ifModule mod_headers.c>
 Header set X-Robots-Tag "noindex, nofollow"
 </ifModule>
 }}}

 Problem is, this might interfere with an already existing `.htaccess`
 file. Then, this snippet could possibly be appended.

 Another problem is that some hosts disallow header manipulation via
 .htaccess (and a lot more).

 So maybe back to the first idea.

 Lastly, how is it not in line with the documentation? As per [3548] it
 outputs `noindex, nofollow` as html meta tag (see `function noindex()`).
 Isn't this correct? Did I miss something?

-- 
Ticket URL: <https://core.trac.wordpress.org/ticket/43590#comment:10>
WordPress Trac <https://core.trac.wordpress.org/>
WordPress publishing platform