X-Robots-Tag: noindex & nofollow
·
Our indexing bots use the X-Robots-Tag in the HTTP page header
·
Our indexing bots will not included pages in the search that are tagged with noindex
·
Our indexing bots will not follow links in pages that are tagged as nofollow
·
The nofollow part of the X-Robots-Tag is out of date and you should be using the Anchor Tag NoFollow instead.
Examples:
X-Robots-Tag: noindex, nofollow
X-Robots-Tag: noindex
X-Robots-Tag: nofollow