Google releases a full list of IP addresses used by Googlebot for Crawling - Infidigit (2024)

• Published on April 15, 2024

• 

  Reviewed By

  Ankit Thakkar

  Ankit Thakkar is a Group Head here, at Infidigit. He is associated with us for the last 6 years. He takes care of more than 40+ client deliveries, manages a team of more than 60 people and also takes care for product delivery at Infidigit. With his SEO strategies, he has won more than 10 awards in the SEO space. Ankit has always loved taking up new challenges and is focused on managing tasks efficiently and effectively. Helpful and Hardworking are a few of his skills that we love. When not working, he loves reading books, listening to music and watching series.

  

  Reviewed By

  Ankit Thakkar

• Reading Time

  2 min read

This is important news for those who use Reverse DNS lookup to verifyGooglebot. Google has published a list of all IP addresses used by Googlebot to crawl and discover websites for Google Search.

Where can I find the IP list?

The list of Googlebot IP addresses has been released in two JSON files.

• JSON 1– This list will help you identify Googlebot by matching the crawler’s IP address with the IP addresses in the JSON file.
• JSON 2– For all other Google crawlers, you can compare the crawler’s IP address against the complete list of Google IP addresses in the JSON file.

JSON can be a bit overwhelming for everyone to understand. Especially when it comes to the JSON files that include the subnet masks. If you’re having difficulties reading the IP address in the JSON file, we recommend calculating the IP ranges usingthis tool.

Why do I need Googlebot IP Addresses?

Crawlers scraping their website is something that no website owner appreciates. This could also lead to sensitive information being compromised. Crawlers with malicious intent can scrape your entire website and steal all of your data. You’re at a high risk of a data breach as a result of this.

However, for SEO, you need to allow the Googlebot agent to crawl your website to reward your SEO efforts in Google Search. Having an entire list of Googlebot IP addresses ensures that you can verify whether the agent requesting to crawl your website is indeed the Googlebot.

There are two methods that site owners can use to verify Googlebot:

1. Manual– This can be done using the command-line tools. You can make use of 1 v 1 lookups. This works for small scale lookups
2. Automated– If you have a large list of IP addresses you want to look up, command-line won’t be the ideal solution. You can use an automated solution to look up all requested IP addresses against a list of verified Googlebot addresses.

This is where the full list of IP addresses used by the Googlebot comes in handy.

Key Takeaway

A website may go down or even crash due to fraudulent and spammy bots sending a large number of requests. Reverse DNS lookup and other methods need to be used from time to time to ensure that only useful bots are crawling your website.

Blocking all the IP addresses isn’t a solution that fits well as you might end up blocking the Googlebot. This will have a significant impact on your Search Engine Optimization performance. You can tell if a request is from Googlebot if you have a list of verified Googlebot IP addresses.