August 26, 2009

A message to site owners

We will be releasing our indexer robot BitSnoop to roam the internets in some not-so-distant future.

You can easily identify him by User-Agent header. He tries to obey robots.txt and not to overload your systems while crawling. Still if you're having problems because of robot - please email him at robot@bitsnoop.com. We receive copies of his emails and will make sure that he behaves.

If you run torrent site and host torrents - let robot know and we will make sure he crawls your site. He is a bit lazy, so it may take some time.

No comments:

Post a Comment