Will you use the new IndexNow protocol?

[MichaelCorfman]

Microsoft and Yandex announced a new initiative on 18 October 2021 named IndexNow (IndexNow.org). It is an open protocol any search engine can participate in that enables websites to easily notify search engines whenever pages are created, updated or deleted.

On November 9th Search Engine Land published the article "Google is testing the IndexNow protocol for sustainability." According to that article, a Google stated "we take a holistic approach to sustainability at Google, including efforts to develop the most efficient and accurate index of the web. We have been carbon neutral since 2007 and we are on pace to be carbon-free by 2030 across both our data centers and campuses. We’re encouraged by work to make web crawling more efficient, and we will be testing the potential benefits of this protocol."

And, yesterday Search Engine Land published a further article discussing adoption of the new protocol: 60,000 websites using Cloudflare turned on IndexNow. According to the article the websites have already sent "about billion Hints for when specific assets on their websites have changed and need to be re-crawled."

There seems to be a certain amount of excitement around this protocol. That's not surprising, since instant indexing is an SEO’s dream when it comes to giving search engines the most updated content on a site.

So, for this week's poll I ask whether you plan to get on the bandwagon and adopt this protocol. Would you adopt it based on the current search engines that make use of it? Would you adopt if if Google did?

Michael

[chaumi]

Well I might be missing something...and probably am....but a billion notifications being sent to search engines every x days....all shouting 'I changed, I changed, me, me, me'...when Google can't handle indexing even now when they have the control and decide whether to go looking or not.....

Sounds like a non goer to me.

I'm probably not getting it somehow. But if it works...and this is how indexing will be 'guaranteed'...yes I think it's fair to say everybody (that wants to be indexed) will be using it!

[MichaelCorfman]

Quoting from the IndexNow.org website:

I submitted 10 thousand URLs today, what will happen next?

If search engines like your URLs and have enough crawl quota for your site, search engines will attempt crawling some or all these URLs.

So, the protocol enables you to tell participating search engines that you think particular pages should be crawled. Search engines will decide themselves whether they think that makes sense to them on a site-by-site basis.

Personally, I like the idea of being able to tell a search engine that particular pages are new or have changes in a way that deserves more attention than randomly crawling a site. I would expect it to result in more effective crawl resource utilization.

Michael

[rmeeuwsen]

Quoting from the IndexNow.org website:
Michael

Yes, that caught my attention too!

They used to say "hey, there's an app for that". So likewise, if some bright developer creates a wordpress plugin to do it for me - I'll definitely do that. But in the meantime I'd like to see someone 'do a report' on it's effectiveness. I'm sure that would be a 'killer' post.

[pokeraussie]

I must be missing something but isn't this essentially what a sitemap is used for?

[MichaelCorfman]

It complements the sitemap in a number of ways. Primarily, it provides a way for a webmaster to specifically identify pages with changes that the webmaster believes warrants a crawl of the page.

Quoting from the IndexNow.org website:

I have a sitemap, do I need IndexNow?

Yes, while sitemaps are an easy way for webmasters to inform search engines about all pages on their sites that are available for crawling, sitemaps are visited by Search Engines infrequently. With IndexNow, webmasters ''don't'' have to wait for search engines to discover and crawl sitemaps but can directly notify search engines of new content.

Personally, I expect the feature would help search engines using it discover new content more quickly. For example, I could see using the feature to notify search engines right away when a new news article is posted. Of course, first I would have to decide I cared enough about the search engines using the feature to do too much. But I could see deploying it over the near term on the one larger casino news site I operate.

Michael

[Former Member 14]

There seems to be a certain amount of excitement around this protocol. That's not surprising, since instant indexing is an SEO’s dream when it comes to giving search engines the most updated content on a site.

Back in the 90's, well before Google was even a blip, one could tweak their site's SEO and watch the SERP results change, albeit not instantly but within literally a couple of hours. Granted the reason this was possible was because there simply wasn't billions of sites.

Anyway, given Google's record of stuffing things up, lets see if they can dodge that bullet, with this project.

[baldidiot]

Pinging SEs when content changes isn't new though, or am I missing something? Eg: Pingomatic has been around for years.

[chaumi]

Pinging SEs when content changes isn't new though, or am I missing something? Eg: Pingomatic has been around for years.

I think the point is that it's purported to be a tool that all SE's 'recognize' and give anything coming from it the relevant weight in their deliberations. Kind of like a 'de facto' one tool that everybody uses.

However you look at it, the likely flaws seem to abound.

[universal4]

Have not had a lot of time to study this but did look into it briefly.

There is already a wordpress plugin out that will handle a portion of this.
https://wordpress.org/plugins/mihdan-index-now/

In some cases, pre-built plugins can do wonders for a site and help with development or certain functions.

In this particular case I wonder how easily it could be compromised, and if so, how easy it would be to detect that was done.

However, writing your own plugin might be a safer approach.

Simplified explanation of what needs to happen:
Determine what pages changed from the database
Post to a script with the urls that changed using the api or json script
Possibly set this up as a cron or scheduled task
Provide the additional ability to perform the function manually if deemed necessary.

I had some brief discussions with folks, and after we read a handful of faq's and reviews at such places as search engine land etc etc etc, on this indexnow script and api, the best we could determine, no more than a few submissions per day seemed along the lines of the most acceptable safe practices.

Some are saying just hit it every time you update a page, and some have said, just make sure they are timed 10 minutes apart.

Also, doing something custom provides the site developer more control of how much overhead the query to determine the array of changed pages takes, and allows them to control the schedule as well. (altho I think the plugin written linked above does have some control of the schedule)

But, most of us here know how fast something like this will be in the list of things that people will immediately try to start abusing and manipulating to their advantage.

I agree with a few others that, the various webmaster tools provided by the engines already do most of this already. If an authorized user of a domain (especially if authorized by a dns text file) submits a changed page, why does Google think they have to rely on an different script to perform the same submission.

But I can also see the value of having a single tool to help with each of the engines at once. This will be interesting as it matures a bit.

Rick
Universal4

[universal4]

For the sake of the argument:

Is an url posted by site A pointing at site B acceptable?

If so, if site A is an authority site, is more weight given to such a submission for Site B from Site A then if it had been submitted by Site B that is not an authority.

As I said, I have not had time to really dig into all the nuances of how this works, so if this part of the argument is completely off base, I apologize, please set the record straight if this can be confirmed or denied.

Rick
Universal4

[buyonaut]

Maybe I'm off topic here, but you can use API to submit your pages to Google and Bing through Rank Math - not so new thing and it works.
It is a pain to set up but once done, works.

https://rankmath.com/blog/google-indexing-api/

[universal4]

Thanks,

Does anyone know if rankmath uses the indexnow site or maybe it was created or either one possibly modeled the submit method on one another?

Using my crystal ball on this subject, seems like Yoast is due for an additional feature, as I do not recall seeing such.

Rick
Universal4

[baldidiot]

Maybe I'm off topic here, but you can use API to submit your pages to Google and Bing through Rank Math - not so new thing and it works.
It is a pain to set up but once done, works.

https://rankmath.com/blog/google-indexing-api/

I did not know this was a thing, that's way better than pinging google!

[universal4]

Yoast has stated, at least so far, they will NOT be adding indexnow functions to their plugin.

https://www.searchenginejournal.com/...ctions/424120/

Joost de Valk, founder of Yoast WordPress SEO plugin

“We’ve not added support to @yoast SEO for it yet, for the simple reason that we don’t see the value yet.”

Joost Acknowledges that IndexNow Benefits Big Sites
He tweeted:
“For large sites like that I definitely see the value.
But to do this and make it a default for 12 million sites, most of which have less than 100 pages, seems overkill.”

Interesting he thinks it may benefit larger sites but he took the stand of not adding the functions.

Seems to me, the larger the site, the more often the larger site owners are buying his premium product.

Rick
Universal4

[baldidiot]

For the sake of the argument:

Is an url posted by site A pointing at site B acceptable?

No... (I think).

Just did a little bit more reading into it and index now isn't just a simple ping a la pingomatic, it also requires a token to verify ownership of the domain. SO a bit like pingomatic 2.0.

Ping URLs are in the format: https://<searchengine>/indexnow?url=url-changed&key=your-key

So the site that initiates the ping is irrelevant really as the ping specifically refers to the changed URL along with a key that is hosted by the site that has been updated.

Ie: You could probably initiate a ping from Site A using Site Bs URL and key but it would only look at Site B and there's no reference to the site it originates from (other than the IP but I'd wager they'd ignore that if they're using keys to verify).

It is interesting that after this was introduced in October and after google said they'd look into it they've suddenly introduced the indexing API (which is a whole different ball game as it requires the site to be verified in GWT rather than just use a self hosted token).

Suggests to me they're not going to use indexnow and instead push their own api system.

[universal4]

Could be, and I think I agree that based on the key usage there is less chance of manipulation, but you never know.

As for the key placed in the site, that does add an extra layer of verification when using the api.

And hopefully any tools, if they are released to help webmaster perform this function, do not rely solely on the html version of the google verification code. Google does NOT provide hml verification code if you use the domain model and not the url model. The ONLY verification acceptable is the dns record.

Rick
Universal4

[baldidiot]

Hang on a minute... the google indexing api is only supposed to be used for specific short lived content types, not general content.

See: https://developers.google.com/search...i/v3/using-api

Currently, the Indexing API can only be used to crawl pages with either JobPosting or BroadcastEvent embedded in a VideoObject.

It looks like rank math have figured out you can use it for other content, but you're not technically supposed to. They also have a warning on the site about it: https://rankmath.com/blog/google-indexing-api/

Note: Google recommends that you use the Indexing API ONLY for Job Posting and Live Streaming websites. During our tests, we found that it works on any type of website with great results and created this plugin for testing.

If you are not sure whether you want to be using this or not, please stop reading and don’t proceed further. As Google recommends, you should not use this indexing API on money websites. Please proceed at your own risk.

[universal4]

Good catch, I had not seen that.

That changes the entire dynamic of the tool.

Rick
Universal4

[wonderpunter]

I have activated it via cloudflare, however little of my traffic is from bing or yahoo so will see