robots.txt

Was ist eine robots.txt?

Eine Robots.txt-Datei dient auf einem Webspace dazu den Suchmaschinen mitzuteilen, welche Seiten nicht durchsucht werden dürfen. Die Robots.txt-Datei beeinflusst das Ranking gar nicht. Sie liegt im Root-Verzeichnis, in dem auch die index.*-Datei liegt. Sie ist ein Defacto-Standard.
Diese Datei sollte aus verschiedenen Gründen existieren:

1. Es gibt nicht nur gute Robots. Manche dienen lediglich dazu einen Informationsklau (z.B. E-Mail-Adressen) durchzuführen.
2. Während eines Relaunches oder der ersten Designumsetzung einer Website ist es sinnvoll die Indexierung generell zu verbieten.
3. Mitgliederbereiche, Profildaten oder private Bereiche, die nicht indexiert werden sollen, können so geschützt werden.

   Standardvorlage

   	User-agent: Name
   	Disallow: [leer = erlaubt, Pfad = verboten; "/" für gesamte Website ab Root]
   	Sitemap: [http-Pfad zu einer Google-Sitemap]
   

   Alle Crawler erlauben

   	# Alle Roboter spidern die Domain
   	User-agent: *
   	Disallow:
   

   Dienst EmailCollector verbieten

   	# Programm EmailCollecter hat keinen Zugriff
   	User-agent: EmailCollector
   	Disallow: /
   

   Böse Robots verbieten

   Abakus-Internet-Marketing

	User-agent: grub-client
	Disallow: /

	User-agent: grub
	Disallow: /

	User-agent: looksmart
	Disallow: /

	User-agent: WebZip
	Disallow: /

	User-agent: larbin
	Disallow: /

	User-agent: b2w/0.1
	Disallow: /

	User-agent: psbot
	Disallow: /

	User-agent: Python-urllib
	Disallow: /

	User-agent: NetMechanic
	Disallow: /

	User-agent: URL_Spider_Pro
	Disallow: /

	User-agent: CherryPicker
	Disallow: /

	User-agent: EmailCollector
	Disallow: /

	User-agent: EmailSiphon
	Disallow: /

	User-agent: WebBandit
	Disallow: /

	User-agent: EmailWolf
	Disallow: /

	User-agent: ExtractorPro
	Disallow: /

	User-agent: CopyRightCheck
	Disallow: /

	User-agent: Crescent
	Disallow: /

	User-agent: SiteSnagger
	Disallow: /

	User-agent: ProWebWalker
	Disallow: /

	User-agent: CheeseBot
	Disallow: /

	User-agent: LNSpiderguy
	Disallow: /

	User-agent: ia_archiver
	Disallow: /

	User-agent: ia_archiver/1.6
	Disallow: /

	User-agent: Teleport
	Disallow: /

	User-agent: TeleportPro
	Disallow: /

	User-agent: MIIxpc
	Disallow: /

	User-agent: Telesoft
	Disallow: /

	User-agent: Website Quester
	Disallow: /

	User-agent: moget/2.1
	Disallow: /

	User-agent: WebZip/4.0
	Disallow: /

	User-agent: WebStripper
	Disallow: /

	User-agent: WebSauger
	Disallow: /

	User-agent: WebCopier
	Disallow: /

	User-agent: NetAnts
	Disallow: /

	User-agent: Mister PiX
	Disallow: /

	User-agent: WebAuto
	Disallow: /

	User-agent: TheNomad
	Disallow: /

	User-agent: WWW-Collector-E
	Disallow: /

	User-agent: RMA
	Disallow: /

	User-agent: libWeb/clsHTTP
	Disallow: /

	User-agent: asterias
	Disallow: /

	User-agent: httplib
	Disallow: /

	User-agent: turingos
	Disallow: /

	User-agent: spanner
	Disallow: /

	User-agent: InfoNaviRobot
	Disallow: /

	User-agent: Harvest/1.5
	Disallow: /

	User-agent: Bullseye/1.0
	Disallow: /

	User-agent: Mozilla/4.0 (compatible; BullsEye; Windows 95)
	Disallow: /

	User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
	Disallow: /

	User-agent: CherryPickerSE/1.0
	Disallow: /

	User-agent: CherryPickerElite/1.0
	Disallow: /

	User-agent: WebBandit/3.50
	Disallow: /

	User-agent: NICErsPRO
	Disallow: /

	User-agent: Microsoft URL Control - 5.01.4511
	Disallow: /

	User-agent: DittoSpyder
	Disallow: /

	User-agent: Foobot
	Disallow: /

	User-agent: WebmasterWorldForumBot
	Disallow: /

	User-agent: SpankBot
	Disallow: /

	User-agent: BotALot
	Disallow: /

	User-agent: lwp-trivial/1.34
	Disallow: /

	User-agent: lwp-trivial
	Disallow: /

	User-agent: BunnySlippers
	Disallow: /

	User-agent: Microsoft URL Control - 6.00.8169
	Disallow: /

	User-agent: URLy Warning
	Disallow: /

	User-agent: Wget/1.6
	Disallow: /

	User-agent: Wget/1.5.3
	Disallow: /

	User-agent: Wget
	Disallow: /

	User-agent: LinkWalker
	Disallow: /

	User-agent: cosmos
	Disallow: /

	User-agent: moget
	Disallow: /

	User-agent: hloader
	Disallow: /

	User-agent: humanlinks
	Disallow: /

	User-agent: LinkextractorPro
	Disallow: /

	User-agent: Offline Explorer
	Disallow: /

	User-agent: Mata Hari
	Disallow: /

	User-agent: LexiBot
	Disallow: /

	User-agent: Web Image Collector
	Disallow: /

	User-agent: The Intraformant
	Disallow: /

	User-agent: True_Robot/1.0
	Disallow: /

	User-agent: True_Robot
	Disallow: /

	User-agent: BlowFish/1.0
	Disallow: /

	User-agent: JennyBot
	Disallow: /

	User-agent: MIIxpc/4.2
	Disallow: /

	User-agent: BuiltBotTough
	Disallow: /

	User-agent: ProPowerBot/2.14
	Disallow: /

	User-agent: BackDoorBot/1.0
	Disallow: /

	User-agent: toCrawl/UrlDispatcher
	Disallow: /

	User-agent: WebEnhancer
	Disallow: /

	User-agent: suzuran
	Disallow: /

	User-agent: VCI WebViewer VCI WebViewer Win32
	Disallow: /

	User-agent: VCI
	Disallow: /

	User-agent: Szukacz/1.4 
	Disallow: /

	User-agent: QueryN Metasearch
	Disallow: /

	User-agent: Openfind data gathere
	Disallow: /

	User-agent: Openfind 
	Disallow: /

	User-agent: Xenu's Link Sleuth 1.1c
	Disallow: /

	User-agent: Xenu's
	Disallow: /

	User-agent: Zeus
	Disallow: /

	User-agent: RepoMonkey Bait & Tackle/v1.01
	Disallow: /

	User-agent: RepoMonkey
	Disallow: /

	User-agent: Microsoft URL Control
	Disallow: /

	User-agent: Openbot
	Disallow: /

	User-agent: URL Control
	Disallow: /

	User-agent: Zeus Link Scout
	Disallow: /

	User-agent: Zeus 32297 Webster Pro V2.9 Win32
	Disallow: /

	User-agent: Webster Pro
	Disallow: /

	User-agent: EroCrawler
	Disallow: /

	User-agent: LinkScan/8.1a Unix
	Disallow: /

	User-agent: Keyword Density/0.9
	Disallow: /

	User-agent: Kenjin Spider
	Disallow: /

	User-agent: Iron33/1.0.2
	Disallow: /

	User-agent: Bookmark search tool
	Disallow: /

	User-agent: GetRight/4.2
	Disallow: /

	User-agent: FairAd Client
	Disallow: /

	User-agent: Gaisbot
	Disallow: /

	User-agent: Aqua_Products
	Disallow: /

	User-agent: Radiation Retriever 1.1
	Disallow: /

	User-agent: Flaming AttackBot
	Disallow: /

	User-agent: Oracle Ultra Search
	Disallow: /

	User-agent: MSIECrawler
	Disallow: /

	User-agent: PerMan
	Disallow: /

	User-agent: searchpreview
	Disallow: /

Crawler Bot-Liste

Forumbeitrag OpenX.org

	Google 72.14.199.*
	AdsBot-Google (+http://www.google.com/adsbot.html)

	Walhello 80.60.35.*
	appie 1.1 (www.walhello.com)

	Walhello 81.205.39.*
	appie 1.1 (www.walhello.com)

	Walhello 84.104.216.*
	appie 1.1 (www.walhello.com)

	Walhello 84.104.217.*
	appie 1.1 ( www.walhello.com)

	Walhello 84.104.39.*
	appie 1.1 (www.walhello.com)

	Walhello 84.104.40.*
	appie 1.1 (www.walhello.com)

	Walhello 84.104.43.*
	appie 1.1 (www.walhello.com)

	Brandimensions 204.92.59.*
	BDFetch

	Brandimensions 216.183.91.*
	BDFetch

	Brandimensions 64.26.160.*
	BDFetch

	Brandimensions 70.25.237.*
	BDFetch

	Brandimensions 72.14.163.*
	BDFetch

	Brandimensions 74.97.49.*
	BDFetch

	Intelliseek 64.158.138.*
	Blogpulse (info@blogpulse.com)

	Commission Junction 216.34.209.*
	CJNetworkQuality; http://www.cj.com/networkquality

	Commission Junction 64.70.54.*
	CJNetworkQuality; http://www.cj.com/networkquality

	Google 84.58.171.*
	CSCrawler/http://www.kde.cs.uni-kassel.de/lehre/ss2005/googlespam/crawler.html RPT-HTTPClient/0.3-3

	Yahoo! 202.93.76.*
	DoCoMo/2.0/SO502i (compatible; Mozilla 4.0; MSIE 6.0; yahooseeker-jp-mobile AT Yahoo!JAPAN)

	Envolk 98.173.26.*
	envolk/1.7 (+http://www.envolk.com/envolkspiderinfo.html)

	Envolk 68.108.249.*
	envolk/1.7 (+http://www.envolk.com/envolkspiderinfo.php)

	Envolk 68.6.111.*
	envolk/1.7 (+http://www.envolk.com/envolkspiderinfo.php)

	Envolk 70.169.191.*
	envolk/1.7 (+http://www.envolk.com/envolkspiderinfo.php)

	Envolk 68.6.101.*
	envolk[ITS]spider/1.6 (+http://www.envolk.com/envolkspider.html)

	Envolk 71.102.140.*
	envolk[ITS]spider/1.6 (+http://www.envolk.com/envolkspider.html)

	Envolk 71.102.160.*
	envolk[ITS]spider/1.6 (+http://www.envolk.com/envolkspider.html)

	FAST 213.3.44.*
	FAST Enterprise Crawler 6 used by EPFL ( raul.rabionet@epfl.ch)

	FAST 217.114.115.*
	FAST Enterprise Crawler 6 used by EPFL ( raul.rabionet@epfl.ch)

	FAST 216.255.229.*
	FAST Enterprise Crawler 6 used by FAST (iverjor (at) fast.no)

	FAST 219.165.118.*
	FAST Enterprise Crawler 6 used by FAST (iverjor (at) fast.no)

	FAST 12.32.25.*
	FAST Enterprise Crawler 6 used by GDSN (e_gfc_it_gdsn-fast@gdsn-home.net )

	FAST 193.252.242.*
	FAST Enterprise Crawler 6 used by Pages Jaunes (crawladmin@gmail.com )

	FAST 81.93.164.*
	FAST Enterprise Crawler 6 used by Schibsted (webcrawl@schibstedsok.no )

	FAST 202.176.218.*
	FAST Enterprise Crawler 6 used by Singapore Press Holdings ( crawler@sphsearch.sg)

	FAST 202.176.220.*
	FAST Enterprise Crawler 6 used by Singapore Press Holdings ( crawler@sphsearch.sg)

	FAST 64.151.114.*
	FAST Enterprise Crawler 6 used by TauMed ( crawler@taumed.com)

	FAST 131.165.88.*
	FAST Enterprise Crawler 6 used by Virk.dk - udvikling ( thomas.bentzen@capgemini.com)

	FAST 69.25.71.*
	FAST Enterprise Crawler/6.4 (helpdesk at fast.no)

	Findexa 80.239.38.*
	Findexa Crawler ( http://www.findexa.no/gulesider/article26548.ece)

	Gigablast 38.114.104.*
	Gigabot/1.0

	Gigablast 63.225.19.*
	Gigabot/1.0

	Gigablast 64.139.94.*
	Gigabot/1.0

	Gigablast 70.90.210.*
	Gigabot/1.0

	Gigablast 74.94.177.*
	GigaBot/1.0

	Gigablast 66.154.102.*
	Gigabot/2.0

	Gigablast 66.231.188.*
	Gigabot/2.0 (http://www.gigablast.com/spider.html)

	Gigablast 66.231.189.*
	Gigabot/2.0 (http://www.gigablast.com/spider.html)

	Gigablast 131.161.247.*
	Gigabot/2.0/gigablast.com/spider.html

	Gigablast 66.154.103.*
	Gigabot/2.0/gigablast.com/spider.html

	Google 71.162.124.*
	Google Spider

	Google 66.249.73.*
	Googlebot-Image/1.0

	Google 125.177.35.*
	Googlebot/2.1 (+http://www.google.com/bot.html)

	Google 207.44.196.*
	Googlebot/2.1 (+http://www.google.com/bot.html)

	Google 222.238.81.*
	Googlebot/2.1 (+http://www.google.com/bot.html)

	Google 66.249.64.*
	Googlebot/2.1 (+http://www.google.com/bot.html)

	Google 66.249.71.*
	Googlebot/2.1 (+http://www.google.com/bot.html)

	Google 65.57.245.*
	gsa-crawler (Enterprise; GIX-02530; enterprise-training@google.com )

	Google 64.233.178.*
	gzip(gfe) (via translate.google.com)

	Google 141.51.167.*
	Hi! I'm CsCrawler, my homepage: http://www.kde.cs.uni-kassel.de/lehre/ss20...am/crawler.html

	Google 84.58.174.*
	Hi! I'm CsCrawler, my homepage: http://www.kde.cs.uni-kassel.de/lehre/ss20...am/crawler.html

	Google 84.58.96.*
	Hi! I'm CsCrawler, my homepage: http://www.kde.cs.uni-kassel.de/lehre/ss20...am/crawler.html

	Yahoo! 218.132.56.*
	larbin_2.6.3 (ewaina_2003@email.yahoo.com)

	Yahoo! 213.91.216.*
	larbin_2.6.3 (vesso1221yahoo.com)

	SEVENtwentyfour 209.167.50.*
	LinkWalker

	Lycos 209.202.205.*
	Lycos_Spider_(modspider)

	Lycos 209.202.229.*
	Lycos_Spider_(modspider)

	Lycos 209.202.248.*
	Lycos_Spider_(modspider)

	Google 66.249.65.*
	Mediapartners-Google/2.1

	Google 66.249.70.*
	Mediapartners-Google/2.1

	Google 66.249.72.*
	Mediapartners-Google/2.1

	MIVA 66.150.55.*
	Miva (AlgoFeedback@miva.com)

	Google 64.178.97.*
	Mozilla 4.0(compatible; Googlebot/5.0; +http://www.google.com/googlebot)

	Google 24.7.232.*
	Mozilla/4.0 (compatible; GoogleToolbar 4.0.1020.2544-big; Windows XP 5.1; MSIE 6.0.2900.2180)

	Google 82.47.1.*
	Mozilla/4.0 (compatible; GoogleToolbar 4.0.1020.2544-big; Windows XP 5.1; MSIE 6.0.2900.2180)

	MSN 68.193.180.*
	Mozilla/4.0 (compatible; MSIE 6.0; MSN 2.5; Windows 98; MSIECrawler)

	Yahoo! 60.10.2.*
	Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1) (m_pupil@yahoo.com.cn)

	LookSmart 64.242.88.*
	Mozilla/4.0 compatible ZyBorg/1.0 Dead Link Checker ( wn.dlc@looksmart.net; http://www.WISEnutbot.com)

	Google 66.249.66.*
	Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

	Google 66.249.67.*
	Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

	Google 85.114.130.*
	Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

	Google 66.230.157.*
	Mozilla/5.0 (compatible; googlebot/2.1; +http://www.google.com/bot.html) (googlebot@google.com)

	Yahoo! 125.22.91.*
	Mozilla/5.0 (compatible; heritrix/1.6.0 +http://www.yahoo.com)

	Yahoo! 202.160.178.*
	Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html)

	Yahoo! 202.160.181.*
	Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html)

	Yahoo! 202.160.185.*
	Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html )

	Yahoo! 202.160.179.*
	Mozilla/5.0 (compatible; Yahoo! Slurp/si-emb; http://help.yahoo.com/help/us/ysearch/slurp)

	Yahoo! 66.228.165.*
	Mozilla/5.0 (compatible; Yahoo! Slurp/si-emb; http://help.yahoo.com/help/us/ysearch/slurp)

	Yahoo! 202.160.180.*
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

	Yahoo! 202.160.183.*
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

	Yahoo! 217.11.235.*
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

	Yahoo! 66.196.101.*
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

	Yahoo! 66.196.65.*
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

	Yahoo! 66.196.77.*
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

	Yahoo! 66.196.90.*
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

	Yahoo! 66.196.91.*
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

	Yahoo! 66.196.92.*
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

	Yahoo! 66.228.167.*
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

	Yahoo! 68.142.246.*
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

	Yahoo! 68.142.249.*
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

	Yahoo! 68.142.250.*
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

	Yahoo! 68.142.251.*
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

	Yahoo! 68.180.251.*
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

	Yahoo! 72.30.101.*
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

	Yahoo! 72.30.102.*
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

	Yahoo! 72.30.103.*
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

	Yahoo! 72.30.104.*
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

	Yahoo! 72.30.107.*
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

	Yahoo! 72.30.110.*
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

	Yahoo! 72.30.111.*
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

	Yahoo! 72.30.128.*
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

	Yahoo! 72.30.129.*
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

	Yahoo! 72.30.131.*
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

	Yahoo! 72.30.132.*
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

	Yahoo! 72.30.133.*
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

	Yahoo! 72.30.134.*
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

	Yahoo! 72.30.135.*
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

	Yahoo! 72.30.177.*
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

	Yahoo! 72.30.214.*
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

	Yahoo! 72.30.215.*
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

	Yahoo! 72.30.216.*
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

	Yahoo! 72.30.221.*
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

	Yahoo! 72.30.226.*
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

	Yahoo! 72.30.252.*
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

	Yahoo! 72.30.54.*
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

	Yahoo! 72.30.56.*
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

	Yahoo! 72.30.61.*
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

	Yahoo! 72.30.81.*
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

	Yahoo! 72.30.97.*
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

	Yahoo! 72.30.98.*
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

	Yahoo! 72.30.99.*
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

	Yahoo! 74.6.130.*
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

	Yahoo! 74.6.131.*
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

	Yahoo! 74.6.17.*
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

	Yahoo! 74.6.18.*
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

	Yahoo! 74.6.19.*
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

	Yahoo! 74.6.20.*
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

	Yahoo! 74.6.22.*
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

	Yahoo! 74.6.23.*
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

	Yahoo! 74.6.24.*
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

	Yahoo! 74.6.25.*
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

	Yahoo! 74.6.26.*
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

	Yahoo! 74.6.27.*
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

	Yahoo! 74.6.28.*
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

	Yahoo! 74.6.29.*
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

	Yahoo! 74.6.65.*
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

	Yahoo! 74.6.66.*
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

	Yahoo! 74.6.67.*
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

	Yahoo! 74.6.68.*
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

	Yahoo! 74.6.69.*
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

	Yahoo! 74.6.70.*
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

	Yahoo! 74.6.71.*
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

	Yahoo! 74.6.72.*
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

	Yahoo! 74.6.73.*
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

	Yahoo! 74.6.74.*
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

	Yahoo! 74.6.75.*
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

	Yahoo! 74.6.76.*
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

	Yahoo! 74.6.8.*
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

	Yahoo! 74.6.85.*
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

	Yahoo! 74.6.86.*
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

	Yahoo! 74.6.87.*
	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

	Google 130.203.146.*
	Mozilla/5.0 (Windows; Mozilla 4.0(compatible; Googlebot/5.0; +http://www.google.com/googlebot); en-US;)

	Google 80.131.221.*
	Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.8.0.1) Gecko/20060111 googlebot 2.1

	Google 84.188.206.*
	Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.8.0.3) Gecko/20060426 Googlebot 2.1

	Yahoo! 209.131.61.*
	Mozilla/5.0 (Yahoo-Test/4.0 mailto:vertical-crawl-support@yahoo-inc.com )

	Yahoo! 202.165.97.*
	Mozilla/5.0 (Yahoo-Test/4.0; mailto:vertical-crawl-support@yahoo-inc.com )

	Yahoo! 203.209.241.*
	Mozilla/5.0 (Yahoo-Test/4.0; mailto:vertical-crawl-support@yahoo-inc.com )

	Yahoo! 209.73.172.*
	Mozilla/5.0 (Yahoo-Test/4.0; mailto:vertical-crawl-support@yahoo-inc.com )

	Yahoo! 209.73.174.*
	Mozilla/5.0 (Yahoo-Test/4.0; mailto:vertical-crawl-support@yahoo-inc.com )

	MSN 65.54.112.*
	msnbot-Products/1.0 (+http://search.msn.com/msnbot.htm)

	MSN 65.55.214.*
	msnbot-Products/1.0 (+http://search.msn.com/msnbot.htm)

	MSN 65.55.216.*
	msnbot-Products/1.0 (+http://search.msn.com/msnbot.htm)

	MSN 65.54.164.*
	msnbot/0.9 (+http://search.msn.com/msnbot.htm)

	MSN 65.55.212.*
	msnbot/0.9 (+http://search.msn.com/msnbot.htm)

	MSN 65.55.213.*
	msnbot/0.9 (+http://search.msn.com/msnbot.htm)

	MSN 131.107.0.*
	msnbot/1.0 (+http://search.msn.com/msnbot.htm)

	MSN 202.96.51.*
	msnbot/1.0 (+http://search.msn.com/msnbot.htm)

	MSN 207.46.89.*
	msnbot/1.0 (+http://search.msn.com/msnbot.htm)

	MSN 207.46.98.*
	msnbot/1.0 (+http://search.msn.com/msnbot.htm)

	MSN 207.68.146.*
	msnbot/1.0 (+http://search.msn.com/msnbot.htm)

	MSN 207.68.154.*
	msnbot/1.0 (+http://search.msn.com/msnbot.htm)

	MSN 207.68.157.*
	msnbot/1.0 (+http://search.msn.com/msnbot.htm)

	MSN 207.68.188.*
	msnbot/1.0 (+http://search.msn.com/msnbot.htm)

	MSN 219.141.219.*
	msnbot/1.0 (+http://search.msn.com/msnbot.htm)

	MSN 219.142.53.*
	msnbot/1.0 (+http://search.msn.com/msnbot.htm)

	MSN 64.4.8.*
	msnbot/1.0 (+http://search.msn.com/msnbot.htm)

	MSN 65.54.165.*
	msnbot/1.0 (+http://search.msn.com/msnbot.htm)

	MSN 65.54.188.*
	msnbot/1.0 (+http://search.msn.com/msnbot.htm)

	MSN 65.55.208.*
	msnbot/1.0 (+http://search.msn.com/msnbot.htm)

	MSN 65.55.209.*
	msnbot/1.0 (+http://search.msn.com/msnbot.htm)

	MSN 65.55.210.*
	msnbot/1.0 (+http://search.msn.com/msnbot.htm)

	MSN 65.55.215.*
	msnbot/1.0 (+http://search.msn.com/msnbot.htm)

	MSN 65.55.233.*
	msnbot/1.0 (+http://search.msn.com/msnbot.htm)

	MSN 65.55.234.*
	msnbot/1.0 (+http://search.msn.com/msnbot.htm)

	MSN 65.55.235.*
	msnbot/1.0 (+http://search.msn.com/msnbot.htm)

	MSN 65.55.246.*
	msnbot/1.0 (+http://search.msn.com/msnbot.htm)

	MSN 65.55.250.*
	msnbot/1.0 (+http://search.msn.com/msnbot.htm)

	MSN 65.55.252.*
	msnbot/1.0 (+http://search.msn.com/msnbot.htm)

	MSN 65.54.158.*
	MSNPTC/1.0

	Yahoo! 209.191.82.*
	Nokia6682/2.0 (3.01.1) (compatible;YahooSeeker/M1A1-R2D2; http://help.yahoo.com/help/us/ysearch/craw...awling-01.html)

	Yahoo! 209.191.123.*
	Nokia6682/2.0 (3.01.1) (compatible;YahooSeeker/M1A1-R2D2; http://help.yahoo.com/help/us/ysearch/craw...awling-01.html)

	Yahoo! 69.147.79.*
	Nokia6682/2.0 (3.01.1) (compatible;YahooSeeker/M1A1-R2D2; http://help.yahoo.com/help/us/ysearch/craw...awling-01.html)

	NameProtect 12.175.0.*
	NPBot/3 (NPBot; http://www.nameprotect.com; npbot@nameprotect.com)

	Yahoo! 70.187.130.*
	nutchsearch/Nutch-0.9 (Nutch Search 1.0; herceg_novi at yahoo dot com)

	PrivacyFinder 128.2.220.*
	PrivacyFinder/1.1

	AltaVista 216.145.49.*
	Scooter/3.3

	AltaVista 66.94.232.*
	Scooter/3.3

	Yahoo! 209.249.86.*
	WebCrawler ( pcp071098@yahoo.com)

	Yahoo! 66.180.173.*
	WebIndexer/1-dev (Web Indexer; mailto://webindexerv1@yahoo.com; webindexerv1@yahoo.com)

	Yahoo! 203.141.52.*
	Y!J-BSC/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)

	Yahoo! 211.14.11.*
	Y!J-PSC/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)

	Yahoo! 211.14.8.*
	Y!J/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html )

	Yahoo! 209.191.83.*
	Yahoo-Blogs/v3.9 (http://help.yahoo.com/help/us/ysearch/crawling/crawling-02.html )

	Yahoo! 68.142.195.*
	Yahoo-Blogs/v3.9 (http://help.yahoo.com/help/us/ysearch/crawling/crawling-02.html )

	Yahoo! 206.190.43.*
	Yahoo-MMAudVid/1.0 (mms dash mmaudvidcrawler dash support at yahoo dash inc dot com)

	Yahoo! 68.142.210.*
	Yahoo-MMAudVid/1.0 (mms dash mmaudvidcrawler dash support at yahoo dash inc dot com)

	Yahoo! 209.191.113.*
	Yahoo-MMAudVid/2.0(mms dash mm aud vid crawler dash support at yahoo dash inc.com ;)

	Yahoo! 216.39.60.*
	Yahoo-MMAudVid/2.0(mms dash mm aud vid crawler dash support at yahoo dash inc.com ;)

	Yahoo! 66.163.174.*
	Yahoo-MMAudVid/2.0(mms dash mm aud vid crawler dash support at yahoo dash inc.com ;)

	Yahoo! 66.163.175.*
	Yahoo-MMAudVid/2.0(mms dash mm aud vid crawler dash support at yahoo dash inc.com ;)

	Yahoo! 69.147.82.*
	Yahoo-MMAudVid/2.0(mms dash mm aud vid crawler dash support at yahoo dash inc.com ;)

	Yahoo! 209.191.65.*
	Yahoo-MMCrawler/3.x (mms dash mmcrawler dash support at yahoo dash inc dot com)

	Yahoo! 66.94.238.*
	Yahoo-MMCrawler/3.x (mms dash mmcrawler dash support at yahoo dash inc dot com)

	Yahoo! 68.142.212.*
	Yahoo-MMCrawler/3.x (mms dash mmcrawler dash support at yahoo dash inc dot com)

	Yahoo! 209.131.41.*
	YahooFeedSeeker Testing/2.0 (compatible; Mozilla 4.0; MSIE 5.5; http://publisher.yahoo.com/rssguide )

	Yahoo! 216.39.58.*
	YahooFeedSeeker Testing/2.0 (compatible; Mozilla 4.0; MSIE 5.5; http://publisher.yahoo.com/rssguide)

	Yahoo! 66.94.237.*
	YahooFeedSeeker Testing/2.0 (compatible; Mozilla 4.0; MSIE 5.5 ; http://publisher.yahoo.com/rssguide)

	Yahoo! 216.109.121.*
	YahooFeedSeeker/1.0 (compatible; Mozilla 4.0; MSIE 5.5; http://my.yahoo.com/s/publishers.html)

	Yahoo! 66.218.65.*
	YahooFeedSeeker/1.0 (compatible; Mozilla 4.0; MSIE 5.5; http://publisher.yahoo.com/rssguide)

	Yahoo! 209.191.126.*
	YahooFeedSeeker/2.0 (compatible; Mozilla 4.0; MSIE 5.5; http://publisher.yahoo.com/rssguide)

	Yahoo! 216.145.53.*
	YahooFeedSeeker/2.0 (compatible; Mozilla 4.0; MSIE 5.5; http://publisher.yahoo.com/rssguide)

	Yahoo! 66.163.187.*
	YahooFeedSeeker/2.0 (compatible; Mozilla 4.0; MSIE 5.5; http://publisher.yahoo.com/rssguide )

	Yahoo! 66.94.229.*
	YahooFeedSeeker/2.0 (compatible; Mozilla 4.0; MSIE 5.5; http://publisher.yahoo.com/rssguide)

	Yahoo! 216.39.51.*
	YahooSeeker/1.2 ( yahooseeker at yahoo-inc dot com ;)

	Yahoo! 68.142.230.*
	YahooSeeker/1.2 ( yahooseeker at yahoo-inc dot com ; )

	Yahoo! 64.157.138.*
	YahooSeeker/CafeKelsa-dev

	Yahoo! 202.46.19.*
	yahooseeker/ms-dev(compatible;Nokia-WAP-Toolkit/2.0;freebsd) 

Wiki-Datei des Artikels herunterladen