#

#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#

robots.txt
This file is to prevent the crawling and indexing of certain parts
of your site by web crawlers and spiders run by sites like Yahoo!
and Google. By telling these "robots" where not to go on your site,
you save bandwidth and server resources.
This file will be ignored unless it is at the root of your host:
Used:
http://example.com/robots.txt
Ignored: http://example.com/site/robots.txt
For more information about the robots.txt standard, see:
http://www.robotstxt.org/wc/robots.html
For syntax checking, see:
http://www.sxw.org.uk/computing/robots/check.html

User-agent: *
Crawl-delay: 10
# Begin block Bad-Robots from robots.txt
User-agent: asterias
Disallow:/
User-agent: BackDoorBot/1.0
Disallow:/
User-agent: Black Hole
Disallow:/
User-agent: BlowFish/1.0
Disallow:/
User-agent: BotALot
Disallow:/
User-agent: BuiltBotTough
Disallow:/
User-agent: Bullseye/1.0
Disallow:/
User-agent: BunnySlippers
Disallow:/
User-agent: Cegbfeieh
Disallow:/
User-agent: CheeseBot
Disallow:/
User-agent: CherryPicker
Disallow:/
User-agent: CherryPickerElite/1.0
Disallow:/
User-agent: CherryPickerSE/1.0
Disallow:/
User-agent: CopyRightCheck
Disallow:/
User-agent: cosmos
Disallow:/
User-agent: Crescent
Disallow:/
User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
Disallow:/
User-agent: DittoSpyder
Disallow:/
User-agent: EmailCollector

Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:

EmailSiphon
EmailWolf
EroCrawler
ExtractorPro
Foobot
Harvest/1.5
hloader
httplib
humanlinks
ia_archiver
InfoNaviRobot
JennyBot
Kenjin Spider
Keyword Density/0.9
LexiBot
libWeb/clsHTTP
LinkextractorPro
LinkScan/8.1a Unix
LinkWalker
LNSpiderguy
lwp-trivial
lwp-trivial/1.34
Mata Hari
Microsoft URL Control - 5.01.4511
Microsoft URL Control - 6.00.8169
MIIxpc
MIIxpc/4.2
Mister PiX
moget
moget/2.1

Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:

mozilla/4
Mozilla/4.0 (compatible; BullsEye; Windows 95)
Mozilla/4.0 (compatible; MSIE 4.0; Windows 95)
Mozilla/4.0 (compatible; MSIE 4.0; Windows 98)
Mozilla/4.0 (compatible; MSIE 4.0; Windows NT)
Mozilla/4.0 (compatible; MSIE 4.0; Windows XP)
Mozilla/4.0 (compatible; MSIE 4.0; Windows 2000)
Mozilla/4.0 (compatible; MSIE 4.0; Windows ME)
mozilla/5
NetAnts
NICErsPRO
Offline Explorer
Openfind
Openfind data gathere
ProPowerBot/2.14
ProWebWalker
QueryN Metasearch
RepoMonkey
RepoMonkey Bait & Tackle/v1.01
RMA
SiteSnagger
SpankBot
spanner
suzuran
Szukacz/1.4
Teleport
TeleportPro
Telesoft
The Intraformant
TheNomad

Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:
Disallow:/
User-agent:

TightTwatBot
Titan
toCrawl/UrlDispatcher
True_Robot
True_Robot/1.0
turingos
URLy Warning
VCI
VCI WebViewer VCI WebViewer Win32
Web Image Collector
WebAuto
WebBandit
WebBandit/3.50
WebCopier
WebEnhancer
WebmasterWorldForumBot
WebSauger
Website Quester
Webster Pro
WebStripper
WebZip
WebZip/4.0
Wget
Wget/1.5.3
Wget/1.6
WWW-Collector-E
Xenu's
Xenu's Link Sleuth 1.1c
Zeus
Zeus 32297 Webster Pro V2.9 Win32

Disallow:/
# SEO-related bots
User-agent: rogerbot
Disallow:/
User-agent: mj12bot
Disallow:/
User-agent: dotbot
Disallow:/
User-agent: ahrefsbot
Disallow:/