How to read a new crawling report in the Google Search Console

I have been analyzing access logs for a long time. Knowing where, when and how often Googlebot goes is the key to SEO success, especially for large sites. Until recently, Google was stingy on this information, and we only learned how quickly it downloaded one URL (the key thing), how many events happened per day, and how much data it downloaded. You had to prepare your own tool and data for anything more detailed, or use a third-party service. That (partially) changed yesterday.

Google has launched a new version of the crawling report. You can find it (perhaps for now) hidden in the Settings tab in GSC (bottom left :).

So far, I can see the data at 1.11. till this day. The original report showed a maximum of 90 days, according to (so far) current help page, the value does not change.

You can now see:

• overview of requests according to status codes
• an overview of requests by file type
• overview by purpose
• by Googlebot type

What to read from the new data and what to do based on the data?

Overview of requests according to status codes

A very valuable report, for which you must, however, know the platform you are working with. In the ideal world, you should only see codes 200 and possibly 304 (URL unchanged). Anything outside is a good thing to explore.

Here I have to appreciate that Google added a trend chart directly to the preview. If you know that a certain amount of redirects (3O1 or 302) or unavailable (404) URLs belong to your site, then the trend is exactly what you need to track. So as soon as a fluctuation occurs, you have two options.

Open the detail of the status code and, with the help of a filter, find the scenarios that it identifies on the website using the URL part. It is, for example, product detail, product list, filters and the like. Once you find such a scenario, you can solve it further.

The second option is to get the access log directly from the server and discover it right there after discovering the trend. Easier to read problems from larger data and log analysis tools. Again, the better, the more information your URL contains about the type of content.

Overview of requests by file type

This data is not of great value to me personally. Analyze HTML files via status codes. You have long seen XML maps in another report and there is no different data. Analysis of images with a status code other than 200 has some potential.

Overview by purpose

On the contrary, this report is interesting. If you are doing SEO for a platform that is growing in content, you will be interested in the Discovery type report. It should ideally have an increasing trend and you can check here what new content will fall into the index. Again, ideally through individual content templates on your site, if you can identify them with a URL.

By Googlebot type

This report is more for geeks. But no detailed data here, it won’t show them exactly the version of Chrome that Google uses. Newly, Googlebot should still keep up with the latest version of Chrome for mortals, but in the access locks you will also find visits with a boot with older Chrome.

My personal extras

With the Export feature, you can store key metrics such as the number of requests and page download time in your own spreadsheet. It used to need a bookmarklet in the past. I definitely recommend making a long-term record.

Personally, I have the impression that there is a little less data in the new report. In the original version, I had millions of requests a day for some websites, in the new version, the numbers are an order of magnitude lower. So something had to change in the methodology. Also, Google data actually seem to show somehow less events than actual access logs from the server.

I also hope that significant changes to the data will be part of the notification system that the GSC has. This will make it easier to respond to potential problems.