Do you block ChatGPT from using your website content?

[MichaelCorfman]

OpenAI, the creator of ChatGPT, has published a robots.txt standard that can be used to block the GPTBot from crawling a website. Documentation is provided on the OpenAI website here:

platform.openai.com/docs/gptbot

Here is some of the information published on that page:

GPTBot

GPTBot is OpenAI’s web crawler and can be identified by the following user agent and string.

User agent token: GPTBot
Full user-agent string: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0; +https://openai.com/gptbot)

Usage

Web pages crawled with the GPTBot user agent may potentially be used to improve future models and are filtered to remove sources that require paywall access, are known to gather personally identifiable information (PII), or have text that violates our policies. Allowing GPTBot to access your site can help AI models become more accurate and improve their general capabilities and safety. Below, we also share how to disallow GPTBot from accessing your site.

Disallowing GPTBot

To disallow GPTBot to access your site you can add the GPTBot to your site’s robots.txt:

User-agent: GPTBot
Disallow: /

However, this is only a partial solution of you don't want your site used to train ChatGPT or other large language models. Common Crawl is another source of training material (commoncrawl.org). According to that site, their bot's robots.txt user-agent string is CCBot, so you can stop them from crawling your site by adding the following two lines to your robots.txt file:

User-agent: CCBot
Disallow: /

So, you can use robots.txt to block OpenAI from directly crawling your website to train future versions of ChatGPT and you can use robots.txt to prevent your content from being included in the common crawl.

Search Engine Journal has an article on this topic:

How to Block OpenAI ChatGPT From Using Your Website Content

For this week's poll I ask if you stop either of these crawlers from scrapping your site using robots.txt, or if you plan to, now that you know how. In addition to voting in the poll, I invite you to share your opinion about such crawling of your site, or of crawling in general. And also whether you view it as being important or not.

Michael

[onlinebetdotcom]

I was wondering if openai would follow this. This needs to be a wordpress plugin on its own.

[PaulEchere]

Might be a noobish qustion, but what's so bad about chat gpt crawling my websites?

[wonderpunter]

Might be a noobish qustion, but what's so bad about chat gpt crawling my websites?

Using your hardwork for thier self gain

[NoDepositCasinos]

Might be a noobish qustion, but what's so bad about chat gpt crawling my websites?

I would summarize it this way:

• Loss of traffic: It could use Information from your website without giving you credit. I've read that some plugins will do it, and in that case, it could be a new traffic source, but I'm skeptical.
• You make it easier to copy your content and style.
• There are no incentives to allow it.

[IWcasino]

I could argue that loss of traffic can actually occur DUE TO blocking the bots. IMHO it is madness to block them. You are essentially blocking the access to your site(s) via alternative interfaces (think AI-powered personal assistants, voice assistants, possibly soon the likes of Siri or Alexa etc.). Whereas, by blocking the bots, you are not really gaining anything:
- those who want to copy your content, still can do it using other means / scraping it and then feeding it into AI tools;
- by blocking the bots you are not blocking the plugins, and I doubt ALL the hundreds of available plugins will ever consistently obey robots.txt and with the speed they are being added, I doubt anyone can keep track of all their user agents

[MichaelCorfman]

Evidently many top sites are beginning to block the ChatGPT spider. According to the following article, published in the last week:

Dozens of big brands have blocked GPTBot, OpenAI’s new web crawler

The article states that currently the 15 most popular sites blocking ChatGPT are:

• amazon.com
• quora.com
• nytimes.com
• shutterstock.com
• wikihow.com
• cnn.com
• foursquare.com
• healthline.com
• scribd.com
• businessinsider.com
• reuters.com
• medicalnewstoday.com
• goodhousekeeping.co
• amazon.co.uk
• tumblr.com

Michael

[NoDepositCasinos]

It's an interesting list. I wonder why some chose to block GPTBot but not CCbot.

[TheGooner]

Evidently many top sites are beginning to block the ChatGPT spider. According to the following article, published in the last week:

Dozens of big brands have blocked GPTBot, OpenAI’s new web crawler

The article states that currently the 15 most popular sites blocking ChatGPT are:

• amazon.com

l

Have to laugh at the idea that Amazon think that they have unique content worth blocking bots for ...
Exactly what ground-breaking content do you think they have that needs such protection ?

[chaumi]

the idea that Amazon think that they have unique content worth blocking bots for ...

I think there are AI writers/builders (or elements of them) specifically designed to build Amazon sites/pages. It could be that Amazon (and the others) are not so worried about the concept of someone 'stealing' the odd bit of content, but more about the overall picture where thousands/millions of pages (with AI-enhanced, scraped/improved Amazon descriptions, pros and cons, user comments, etc) are going up per day.

It only takes a small percentage to rank (at those volumes, on a 6000 page site, that could be 6000 visitors a day if there's only one per page), and they're losing money - from the extra affiliate management costs they incur (both the commissions and the admin costs).

And, 'all' they have to do is optimize better than Amazon or work out how best to get 'freshness' boosts, and that visitor figure goes higher (perhaps with no longevity, but then it's just another case of 'make hay before you crash and burn, then do it again' (until the whole thing grinds to a halt, which may never happen).

[eenzoo]

But what sense makes it to block ChatGPT if the database delivered to users do not contains up2date info.
There are no information available from any sources after the September 2021

All published content older than Sep. 2023 has been crawled anyway.

[chaumi]

But what sense makes it to block ChatGPT if the database delivered to users do not contains up2date info.
There are no information available from any sources after the September 2021

All published content older than Sep. 2023 has been crawled anyway.

I'm pretty sure you can send a chat bot/scraper to a page in real time now, by choice. And fairly sure you can do it specifically with ChatGPT.

[Shin Hasegawa]

I am not currently doing that.

Just a question though, is there any benefit to blocking or allowing chatGPT BOT to crawl my pages?

[IWcasino]

Amazon blocking it is madness really

[IWcasino]

Have to laugh at the idea that Amazon think that they have unique content worth blocking bots for ...
Exactly what ground-breaking content do you think they have that needs such protection ?

It's most likely to prevent Amazon affiliates from scraping Amazon and quickly generating their own affiliate pages. It doesn't really stop scraping completely though:

https://twitter.com/IrishWonder/status/1697637247577976856

[IWcasino]

What amazon could do instead is build a plugin for ChatGPT (just like many other brands, especially in travel, did) and benefit from it rather than go all stone age

[TheGooner]

It's most likely to prevent Amazon affiliates from scraping Amazon and quickly generating their own affiliate pages. It doesn't really stop scraping completely though:

https://twitter.com/IrishWonder/status/1697637247577976856

Good find - interesting chat ...

The Chatbot used a scraper that's allowed by Amazon. It's a real tight block then?
Nice coding so that it suggested it and then did it with a simple command to "use another way" ...

Well done chatGPT ... and how exactly is that hurt Amazon anyway ?

[universal4]

Well instead of selling direct, an affiliate might make a quarter which Amazon now has to pay out in a commission.

And they are likely concerned the duplicate content on affiliate sites may well rank batter then their own pages.

Rick
Universal4

[TheGooner]

Hey IWcasino what bot were you chatting to to do that ?

I tried a similar thing with ChatGPT - just to access my own site for lists and got a big fail.

I'm sorry, but I can't directly browse websites in real-time or access up-to-date content from external sites. However, if you provide me with details or specific questions about football (soccer) or other topics, I'll do my best to help based on the information available up to my last training cut-off in September 2021.
If you're interested in current match previews, I'd suggest visiting the website directly or using a web browser. If you have any other questions, feel free to ask!

[IWcasino]

Hey IWcasino what bot were you chatting to to do that ?

I tried a similar thing with ChatGPT - just to access my own site for lists and got a big fail.

You have to have a paid account and plugins enabled - then install WebPilot and maybe some others, I used Scraper in this case but stuff gets added every day, maybe there's something else available now that's even better. Up to 3 plugins can be activated at a time. WebPilot is very smart - it not only accesses the web live (which is what ChatGPT cannot currently do due to a temporary restriction), but it can also browse the sites it accesses, click links etc. However, it does get blocked more than say Scraper - which is why I think I was able to scrape Amazon with Scraper but not with WebPilot. Amazon was actually notoriously difficult to scrape for a long while (namely, links to the actual products - I was unable to collect them with either plugin, neither could I collect them a few years back when I was writing my own scraping scripts), so what they are doing blocking ChatGPT is not out of their character.