[wp-meta] [Making WordPress.org] #398: Block crawlers from tags and other paged results

[Making WordPress.org]

#398: Block crawlers from tags and other paged results
------------------------+---------------------
Reporter:  bpetty       |      Owner:
    Type:  enhancement  |     Status:  new
Priority:  normal       |  Component:  General
Keywords:               |
------------------------+---------------------
 Specifically in regards to the theme and plugin directories, it would be
 helpful if the paginated results for plugins and themes weren't the
 results returned from search engines (rather, the plugin and theme pages
 themselves).

 This is mostly just a problem with tag pages from what I can tell, and
 those can certainly be removed without making any plugins or themes
 unreachable from search crawlers:

 {{{
 Disallow: /themes/tags
 Disallow: /plugins/tags
 }}}

 Those paginated tag results often come up in search results since
 obviously, if you're searching with keywords based on those tags, the tag
 pages themselves contain those keywords in the title and header, while the
 theme or plugin page itself doesn't.

 I would also suggest the popular and newest paginated results, but doing
 these as well might make it impossible for crawlers to reach and index
 many plugin and theme pages...

 {{{
 Disallow: /themes/browse/popular/page
 Disallow: /themes/browse/new/page
 Disallow: /plugins/browse/popular/page
 Disallow: /plugins/browse/new/page
 }}}

 I would like to at least see tags disallowed though.

--
Ticket URL: <https://meta.trac.wordpress.org/ticket/398>
Making WordPress.org <https://meta.trac.wordpress.org/>
Making WordPress.org