Network Unreachable and Robots.txt Unreachable

If you are regularly getting URL unreachable errors like network unreachable and robots.txt file unreachable for Googlebot in Google's Webmaster Tools and you haven't altered anything major like your .htaccess file, CMS, or scripts, then it may be time to find a new web host. According to Google's URL unreachable errors page, these errors are caused by DNS or server issues, which are generally problems with your web host. I had these network unreachable and robots.txt file unreachable errors with my site, and switching web hosts fixed the problem.

Possible causes for URL unreachable errors:

• DNS error
• Server is misconfigured or internal server error
• Server is down
• Server busy or overloaded (a real problem with oversold shared web hosting)
• Dynamic pages are taking too long to respond
• Server is blocking Googlebot (a web host's stop-gap "solution" to their overloaded server problem)

An occasional URL unavailable error is not unusual, but if you are regularly getting them or getting lots of them, then it is a cause for concern. These sort of URL unavailable errors can have serious negative effects on your website. New pages and images will not be indexed by Google and older pages may be dropped from Google's search index, resulting in a drop in Google traffic and frustration for people using Google to search for the information on your site. Since Mediapartners-Google, Google's Adsense robot, is most likely blocked along with Googlebot, your contextual ads will become generic and poorly targeted.

Adding and verifying your website in Google Webmaster Tools is an excellent way to monitor for these sort of serious DNS and server problems because they are generally not going to show up in your web logs. In fact, the only thing that may show up in your web logs in this situation is the absence of Googlebot and the Mediapartners-Google robot.

Note: Although it is an option in Google Webmaster Tools, adding a sitemap is not required to use this free service. Adding a sitemap will do nothing to fix network unreachable and robots.txt file unreachable errors, and you may just get similar errors for your sitemap.