How Google’s Crawl Budget Works – #SEOForLunch Newsletter Issue #19

How Google’s Crawl Budget Works – #SEOForLunch Newsletter Issue #19

Issue # 19
January 17, 2017

Hey, lunch buddies. Our team had some big deliverables this past week and we met them without issue.  Anyone who works agency side has had those long nights and I’m incredibly thankful to have avoided that.

On the personal side, we had a kid-free night where they spent the night at their grandparent’s house.  My wife and I took advantage of it and went out to a nice dinner and even hit up the casino for a few hours.  While I lost, my wife won.  Overall it was a good changeup from the regular routine.  I won’t lie, it helps recharge and replenish the motivation tank.

I hope everyone had as equal if not a better week.


What You Must Know

Article: What Crawl Budget Means For GoogleBot
What’s important?
Gary Illyes posted on the Google Blog details exactly how crawl budget works and how people should keep it in consideration when optimizing their websites.
Nick’s take:
For those with small websites, you don’t likely need to be concerned with crawl budget.  For those that have hundreds of thousands or millions of pages, it’s something you should be aware of.  Ecomm clients should be dealing with their faceted navigation appropriately and everyone should be removing duplicates URLs (not using canonical tags always) to address duplicate content.

It’s also nice to get in writing this:

Q: Can I control Googlebot with the “crawl-delay” directive?
A: The non-standard “crawl-delay” robots.txt directive is not processed by Googlebot.

With that said, please stop putting “crawl-delay” in your robots files!


SEO Tips & Advice

The folks over at Yoast published this killer guide to structured data with Schema.Org

If you’re not aware of what Schema.org is then I highly suggest you read this piece from top to bottom.  Those that are familiar will likely learn a thing or two as well.

For instance, leveraging structured data within AMP implementation was new to me.  Great job Yoast team!


Interesting Reads in Your Spare Time

That’s everything for this week’s folks.  As always, feel free to connect with me, tweet me or hit reply to email me directly. I respond to all emails personally.

Nick

About Nick LeRoy

I'm a husband, father to three and obsessed Vikings fan and SEO Manager at Ovative/group. I also write a weekly SEO newsletter that you can sign up for over here to stay up to date with SEO best practices and industry news.

Entries by Nick LeRoy