There’s nothing more frustrating for a webmaster than having to wait to be crawled. After you’ve perfected your on-site SEO, created some great content, and ensured you have a spider-friendly link structure, all you can do is sit back and wait for the search engine crawlers to come a-knocking. Google will crawl your site at a frequency it determines, and while you can use Google Webmaster Tools to set a crawl rate, that will just limit crawls to a rate below the one you specify to prevent your server from getting hammered. It won’t encourage Googlebot to pay a visit any more frequently if its algorithms decide that there is no need. There are, however, methods you can use to speed up the bots crawl rate, and today we’ll have a look at 4 of them.
Get Links From Frequently Crawled Sites
As with most things Google, it’s the links that count. If your pages are of a sort to encourage links from high-authority sites which are crawled frequently, then any changes to your site are likely to get indexed more quickly. And the best way to get links is …
Have Great Content
Especially frequently-updated, high-quality content that is likely to attract links from other sites. A blog is a great way to ensure that your site is being updated with fresh material regularly (which will help with SEO too). Maintaining a constant flow of new material is the best way to make sure that your site is crawled frequently.
Make Sure Your Server is up to the Job
The above methods will make no difference if your server is down when you are due to be crawled, or drops dead under the strain of being hit by the crawlers. Make sure you have sufficient memory and bandwidth to accommodate both your users and Googlebot.
Use Fetch As Google
Google do provide a means of letting them know that you have new pages up and requesting that they crawl it. In the Google Webmaster Tools, under Health, click Fetch as Google, and submit your URL. There is a limit of 500 URLs per week. This is likely to get you indexed more quickly, but manually submitting every new URL on your site is time-consuming, so this isn’t really a substitute for creating a site that attracts high-quality backlinks and is replete with great content.
If you have any tips to share on increasing the frequency a site is crawled, let us know in the comments, or join in the conversation on Twitter.
