[wp-trac] [WordPress Trac] #43590: Search Engine Visibility option does not work as intended

[WordPress Trac]

#43590: Search Engine Visibility option does not work as intended
----------------------------+-----------------------------
 Reporter:  mamaedler       |      Owner:
     Type:  defect (bug)    |     Status:  new
 Priority:  normal          |  Milestone:  Awaiting Review
Component:  General         |    Version:  4.9.4
 Severity:  normal          |   Keywords:
  Focuses:  administration  |
----------------------------+-----------------------------
 In '''Settings -> Reading''' is an option called "Discourage search
 engines from indexing this site".

 Unfortunately it does not work a intended.

 '''Current behavior'''
 This results in a robots.txt with the following contents:
 {{{
 User-agent: *
 Disallow: /
 }}}

 This is a problem, because the page can still appear in search results in
 some circumstances with the text "No information is available for this
 page." (see attached screenshot). It's because the site's contents are not
 crawled, but the link itself is indexed nevertheless.

 '''Expected behavior'''
 The page shouldn't be listed in search engines at all.

 Google has a help page with the topic
 [https://support.google.com/webmasters/answer/93710?hl=en Block search
 indexing with 'noindex'].

 It states:
 "Important! For the noindex directive to be effective, the page '''must
 not''' be blocked by a robots.txt file. If the page is blocked by a
 robots.txt file, the crawler will never see the noindex directive, and the
 page can still appear in search results, for example if other pages link
 to it."

 In essence, WordPress should return a robots meta tag like this:
 {{{
 <meta name="robots" content="noindex">
 }}}
 and/or return an X-Robots-Tag in the http header like this:
 {{{
 HTTP/1.1 200 OK
 (…)
 X-Robots-Tag: noindex
 (…)
 }}}

 But '''should not block''' access in the first place via robots.txt.

--
Ticket URL: <https://core.trac.wordpress.org/ticket/43590>
WordPress Trac <https://core.trac.wordpress.org/>
WordPress publishing platform