How Google’s Crawl Budget Works – #SEOForLunch Newsletter Issue #19

Share on facebook
Share on twitter

Issue # 19
January 17, 2017

Hey, lunch buddies. Our team had some big deliverables this past week and we met them without issue.  Anyone who works agency side has had those long nights and I’m incredibly thankful to have avoided that.

On the personal side, we had a kid-free night where they spent the night at their grandparent’s house.  My wife and I took advantage of it and went out to a nice dinner and even hit up the casino for a few hours.  While I lost, my wife won.  Overall it was a good changeup from the regular routine.  I won’t lie, it helps recharge and replenish the motivation tank.

I hope everyone had as equal if not a better week.

What You Must Know

Article: What Crawl Budget Means For GoogleBot
What’s important?
Gary Illyes posted on the Google Blog details exactly how crawl budget works and how people should keep it in consideration when optimizing their websites.
Nick’s take:
For those with small websites, you don’t likely need to be concerned with crawl budget.  For those that have hundreds of thousands or millions of pages, it’s something you should be aware of.  Ecomm clients should be dealing with their faceted navigation appropriately and everyone should be removing duplicates URLs (not using canonical tags always) to address duplicate content.

It’s also nice to get in writing this:

Q: Can I control Googlebot with the “crawl-delay” directive?
A: The non-standard “crawl-delay” robots.txt directive is not processed by Googlebot.

With that said, please stop putting “crawl-delay” in your robots files!

SEO Tips & Advice

The folks over at Yoast published this killer guide to structured data with Schema.Org

If you’re not aware of what is then I highly suggest you read this piece from top to bottom.  Those that are familiar will likely learn a thing or two as well.

For instance, leveraging structured data within AMP implementation was new to me.  Great job Yoast team!

Interesting Reads in Your Spare Time

That’s everything for this week’s folks.  As always, feel free to connect with me, tweet me or hit reply to email me directly. I respond to all emails personally.


Nick LeRoy
Nick LeRoy
Nick has 10+ years of SEO consulting experience building and executing strategies for clients ranging from multi-billion dollar consumer packaged goods to fashion entities with hundreds of physical store locations. He also writes the weekly SEO newsletter #SEOForLunch that allows you to stay up to date with industry updates in the amount of time it takes you to eat your lunch.

Make SEO Business Decisions?

The #SEOForLunch newsletter keeps you up to date with SEO industry updates over your lunch break once a week.

Recent Publications