We will be releasing our indexer robot BitSnoop to roam the internets in some not-so-distant future.
You can easily identify him by User-Agent header. He tries to obey robots.txt and not to overload your systems while crawling. Still if you're having problems because of robot - please email him at robot@bitsnoop.com. We receive copies of his emails and will make sure that he behaves.
If you run torrent site and host torrents - let robot know and we will make sure he crawls your site. He is a bit lazy, so it may take some time.
August 26, 2009
Subscribe to:
Post Comments (Atom)
No comments:
Post a Comment