[wp-trac] [WordPress Trac] #50456: Multisite robots.txt files should reference all network XML sitemaps

[WordPress Trac]

#50456: Multisite robots.txt files should reference all network XML sitemaps
----------------------------+-----------------------------
 Reporter:  jonoaldersonwp  |      Owner:  (none)
     Type:  defect (bug)    |     Status:  new
 Priority:  normal          |  Milestone:  Awaiting Review
Component:  Sitemaps        |    Version:
 Severity:  normal          |   Keywords:  seo
  Focuses:                  |
----------------------------+-----------------------------
 https://core.trac.wordpress.org/changeset/48072 adds XML sitemaps to core,
 with the objective of making public URLs more 'discoverable'.

 Part of this discoverability relies on alterations to the site's
 robots.txt file, which add a reference the URL of the sitemap index.

 On multisite setups where sites run in ''subfolders'', this mechanism
 breaks; a domain can only have one robots.txt file at the domain root,
 which means that sub-sites don't expose the location of their sitemap.

 To address this, we should, in all viable cases, add the sitemap URL(s)
 for ''every site in a network'' to the top-level robots.txt file.

 For the sake of completeness, robustness and utility, this should be
 extended to also include multi-site setups on multiple domains/subdomains
 (or in fact, on any setup).

 NB, most consumers support cross-domain XML sitemap references in
 robots.txt files, so this isn't a concern.

 E.g.,

 On a theoretical multi-site setup running across multiple hostnames
 ''and'' folders, I'd expect https://www.example.com/robots.txt to contain
 something like the following:

 {{{
 Sitemap: https://www.example.com/wp-sitemap.xml
 Sitemap: https://www.example.com/sub-site/wp-sitemap.xml
 Sitemap: https://other.example.com/wp-sitemap.xml
 }}}

-- 
Ticket URL: <https://core.trac.wordpress.org/ticket/50456>
WordPress Trac <https://core.trac.wordpress.org/>
WordPress publishing platform