How do you publish the existence of your sitemaps to Google and other search engines?

[MichaelCorfman]

I noticed today that Google announced that the Robot Exclusion Protocol (REP) was submitted to the Internet Engineering Task Force (IETF) as a proposed standard at the beginning of this month on 1 July 2019. Many of us refer to it as robots.txt, since that is the file used to specify the protocol for a specific site. You can see Google's announcement here: Formalizing the Robots Exclusion Protocol Specification.

You can also see a draft of the proposed standard in pdf format here: https://tools.ietf.org/pdf/draft-rep-wg-topic-00.pdf.

At the same time, Google's made another announcement: Google's robots.txt parser is now open source. You can access the source code for Google's robot.text parser on GitHub here: https://github.com/google/robotstxt.

Now, I recently asked folks use a robots.text file (well, just over a year ago) in the following poll: Do you use robots exclusion standard on your sites? So I'm not going to ask that question again (you can still vote in and read that poll if you wish). Thinking about robots.txt once again, I have an entirely different question. One related to the combination of robots.txt and sitemaps.

I'd always assumed that the only way to tell Google about a sitemap was to submit a site map using Google Search Console. And that certainly works well. However, about a year ago I found out there was also another way, and that is to use the sitemap directive within a robots.txt file. Over time I've come to view that as something I always want to do for a variety of reasons. First, other search engine crawlers will find a sitemap that way, and while I certainly don't mind the extra work to specify the sitemap to Google, I have no interest in doing that for every possible search engine. Plus I also use screaming frog, and it understands the robots.txt sitemap directive, making full use of that tool just a little bit easier.

So, for this week's poll I ask how you publish the existence of your sitemaps (if you have any) to Google and to other search engines.

Besides voting the the poll, be sure to share your opinion about Google's recent announcements, and sitemap submission to search engines.

Michael

[ddm]

goog also announced that they will be retiring some unsupported + unpublished rules from robots.txt .. (eg. noindexing) on 1st sept 2019

[PROFRBcom]

Use a module that automatically uploads a new sitemap every 12 hours.

[Mrmork.Jarttu84]

Manually check like once a week from search console/bing that they have indexed the sitemaps and if not I re-add them