How Google’s Crawl Budget Works – #SEOForLunch Newsletter Issue #19
Issue # 19
January 17, 2017
Hey, lunch buddies. Our team had some big deliverables this past week and we met them without issue. Anyone who works agency side has had those long nights and I’m incredibly thankful to have avoided that.
On the personal side, we had a kid-free night where they spent the night at their grandparent’s house. My wife and I took advantage of it and went out to a nice dinner and even hit up the casino for a few hours. While I lost, my wife won. Overall it was a good changeup from the regular routine. I won’t lie, it helps recharge and replenish the motivation tank.
I hope everyone had as equal if not a better week.
What You Must Know
Article: What Crawl Budget Means For GoogleBot
Gary Illyes posted on the Google Blog details exactly how crawl budget works and how people should keep it in consideration when optimizing their websites.
For those with small websites, you don’t likely need to be concerned with crawl budget. For those that have hundreds of thousands or millions of pages, it’s something you should be aware of. Ecomm clients should be dealing with their faceted navigation appropriately and everyone should be removing duplicates URLs (not using canonical tags always) to address duplicate content.
It’s also nice to get in writing this:
Q: Can I control Googlebot with the “crawl-delay” directive?
A: The non-standard “crawl-delay” robots.txt directive is not processed by Googlebot.
With that said, please stop putting “crawl-delay” in your robots files!
SEO Tips & Advice
The folks over at Yoast published this killer guide to structured data with Schema.Org
If you’re not aware of what Schema.org is then I highly suggest you read this piece from top to bottom. Those that are familiar will likely learn a thing or two as well.
For instance, leveraging structured data within AMP implementation was new to me. Great job Yoast team!
Interesting Reads in Your Spare Time
- Google has killed off their Link: command – Barry Schwartz
- Google to send webmaster alerts if mobile first index may impact them – Jennifer Sleg
- First examples of Google’s interstitial penalty – Glenn Gabe
- How to delegate SEO work effectively – Zee Hoffman Jones