Understanding Crawling & Search Queries

[edgarf76]

I am trying to get a better understanding of the crawling rates when I am in webmaster tools. Sometimes my sites get crawled at a much faster rate and sometimes it drops down big time.

What makes a site get crawled faster? Is there anything I can do to make my site get crawled at a faster rate?


Another thing I am trying to get a better understanding of is the number of search queries. My site fluctuates up and down big time on search queries on some of the same keywords but my overall traffic isnt affected, its just which key words people are coming in on, Does anyone know why this is happening? And What Does this mean?

Any Feedback or Suggestions would be appreciated.

[sweetbet]

From what I've read and experienced, sites that update more regularly, are crawled more regularly. I also find that a sitemap helps in getting new content discovered & indexed faster. Why your site might fluctuate in the serps might have something to do with which data center is handling the search query. This is just my guess though.

[edgarf76]

From what I've read and experienced, sites that update more regularly, are crawled more regularly. I also find that a sitemap helps in getting new content discovered & indexed faster. Why your site might fluctuate in the serps might have something to do with which data center is handling the search query. This is just my guess though.

@sweetbet, when you say updated regularly, does 1-2 articles a day considered regular or is it more like 5-10 articles a day?

I also have the google sitemaps plugin for wordpress on my sites so it should be updating them automatically.

This may be an extremely fundamental questions do ask, but what makes a site get crawled faster? Also what is the benefit to the site if it gets crawled faster?

[universal4]

Edgar I think it has to do with what you will average on a site.

In your example, if you plan on adding 5 to 10 articles a week, then 1 to 2 a day is perfect.

If you will be adding 1 or 2 articles that week, space them out a few days.

This gives the spiders the opportunity to see that a visit every few days will likely produce new artcles to crawl.

Rick
Universal4

[Scampi]

When you add a new page, you can use Fetch as Google in Webmaster Tools. This will index it instantly, then make sure your sitemap is in good order. You could use <changefreq> and <priority> tags in your sitemap, setting them to daily and .5 is a nice setting in my opinion.

[edgarf76]

@gamblingjoe, thank you! I am using the google sitemap plugin I think on that I have my posts as a priority over pages.


The webmaster tools technique I have to check out

[slotplayer]

Googles been doing some crazy stuff lately. WMT is showing me 5 404 soft errors from really old pages all of a sudden.

Some say linked from my xml sitemap which is definately not the case.
I'm going out on a guess here and say its something with Godaddy and my htacess file.

[sweetbet]

@sweetbet, when you say updated regularly, does 1-2 articles a day considered regular or is it more like 5-10 articles a day?

I also have the google sitemaps plugin for wordpress on my sites so it should be updating them automatically.

This may be an extremely fundamental questions do ask, but what makes a site get crawled faster? Also what is the benefit to the site if it gets crawled faster?

In terms of regular updates, the more regularly you update your site, the more often your site will be crawled. For example, a site that posts 1-2 new articles per day is being updated regularly, and a site that posts 5-10 articles per day that are spread out during the course of the day is also being updated regularly but will most likely be crawled more often because more content is being added to it more often.