# Aggressive crawlers & scrapers User-agent: FriendlyCrawler User-agent: Orbbot User-agent: PHPCrawl User-agent: Nutch User-agent: magpie-crawler User-agent: Heritrix User-agent: Indy Library User-agent: Go-http-client Disallow: / # SEO & data mining bots User-agent: AhrefsBot User-agent: SemrushBot User-agent: MJ12bot User-agent: BLEXBot User-agent: DotBot User-agent: ZoominfoBot User-agent: MaxPointCrawler User-agent: DataForSeoBot User-agent: seoscanners.net Disallow: / # AI / LLM crawlers User-agent: GPTBot User-agent: ClaudeBot User-agent: Bytespider User-agent: PetalBot User-agent: Barkrowler Disallow: / # Regional / foreign search engines User-agent: YandexBot User-agent: SeznamBot User-agent: Mail.Ru User-agent: coccocbot-image Disallow: / # Social / marketing crawlers User-agent: Amazonbot User-agent: ADmantX User-agent: Buzzbot User-agent: trendictionbot User-agent: rogerbot User-agent: ZumBot User-agent: AlphaBot User-agent: AspiegelBot Disallow: / # Privacy search engines User-agent: DuckDuckBot Disallow: /