robots.txt file

[casionmark]

Hi -

Just setting one of these up for a new site. I have seen the following in another site I have access to. I assume this code is blocking the google bots from crawling the WP admin back end. Is this something that should always be done? Should I include it in my robots.txt?

User-agent: *
Disallow: /wp-admin*
Disallow: /wp-login.php
Disallow: /wp-content*

Also, someone has set up a robots.txt for me and it has the following code:

Crawl-Delay: 20

In Search Console this is marked as an error. Any ideas why?

Thanks,

[Progger]

Not importend,

The Bot stuck on the login,u can block him or ignore simple the "crawling error" reports in GWT.


User-agent: *

Disallow: /wp-admin/
Disallow: /wp-includes/



Regards

[schegolev100]

Hi,

From site on WP recommended close duplicate pages and admin panel from index.

Example robots.txt

User-agent: *

Disallow: /page/*
Disallow: /uncategorized/*
Disallow: /wp-admin/*
Disallow: /wp-admin/admin-ajax.php*
Disallow: /p=*
Disallow: /?*
Disallow: */page/*
Disallow: */attachment/*
Disallow: /?s=
Disallow: /?feed=
Disallow: */feed/
Disallow: */trackback
Disallow: /xmlrpc.php
Disallow: /wp-register.php
Disallow: /wp-login.php
Disallow: /wp-includes

[citizen42]

you may want to remove wp-content/ directory as you may have files directly referenced in there (say images).
wp-includes/ and wp-admin/ should be disallowed, that's correct.

as far as I know googlebot ignores the crawl-delay directive, I'd remove that as well.

also it's a good practice to add the sitemap URL at the end of the robots.txt file.
Sitemap: yousite.com/path-to-sitemap.xml

[RichardV]

don't forget to disallow your affiliate links as well:
Disallow: /visit/*
or whatever other path you're using for it.