Saturday, July 14, 2007

Blogger Blogs have Robot.txt file

Markhutch just informed on Digital Point that Blogger Blog do have Robot.txt support auto-enabled. Though Blogger has not officially came up with any announcement, this is indeed true. You can also check you Robot.txt at your blog URL.

Blogger Robot.txt location

http://yourbloggername.blogspot.com/robot.txt

Just replace yourbloggername with your blogger blog name.

What is Robot.txt means?

Robot .txt is small file which is hosted at root directory in your website. Robot.txt has access to search engine web bots.It directs web-bots and tells whether to index the site or leave some pages. Normally, Robot.txt is in this format

User-agent: googlebot

Disallow: /cgi-bin/

But I found that Blogger Robot.txt is bit different and is shown as below for my site.

User-agent: *Disallow: /search Sitemap: http://techmake.blogspot.com/feeds/posts/default?orderby=updated

Looks that Blogger has one another secret sitemap too. So I don't think we have to add any extra sitemap to webmasters tools from Google, as Google bot automatically picks the sitemap.

But the result ma very after you remove sitemap

Normally, My post gets indexed within 4 days of publishing on Internet. But this time My last post did not got index after I removed my official sitemap. So I have added it again and see the result, it got indexed in a week.

Related Post: How to submit Blogger Sitemap And get every pages indexed in a week