User-agent: * Allow: / Disallow: /api/ Disallow: /app Disallow: /admin Disallow: /login Disallow: /register Disallow: /forgot-password Disallow: /reset-password Disallow: /verify-email Disallow: /profile Disallow: /slots Disallow: /payments Disallow: /static/css/ Disallow: /static/js/ Disallow: /static/fonts/ Disallow: /cdn-cgi/ # ============================================================================ # Search Engines — need CSS/JS for rendering and mobile-friendliness checks # ============================================================================ User-agent: Googlebot Allow: / Disallow: /api/ Disallow: /app Disallow: /admin Disallow: /login Disallow: /register Disallow: /forgot-password Disallow: /reset-password Disallow: /verify-email Disallow: /profile Disallow: /slots Disallow: /payments User-agent: Bingbot Allow: / Disallow: /api/ Disallow: /app Disallow: /admin Disallow: /login Disallow: /register Disallow: /forgot-password Disallow: /reset-password Disallow: /verify-email Disallow: /profile Disallow: /slots Disallow: /payments # ============================================================================ # AI Crawlers — Allowed for content, blocked from assets (recommend and cite) # ============================================================================ User-agent: GPTBot Allow: / Disallow: /api/ Disallow: /app Disallow: /admin Disallow: /login Disallow: /register Disallow: /forgot-password Disallow: /reset-password Disallow: /verify-email Disallow: /profile Disallow: /slots Disallow: /payments Disallow: /static/css/ Disallow: /static/js/ Disallow: /static/fonts/ Disallow: /cdn-cgi/ User-agent: ChatGPT-User Allow: / Disallow: /api/ Disallow: /app Disallow: /admin Disallow: /login Disallow: /register Disallow: /forgot-password Disallow: /reset-password Disallow: /verify-email Disallow: /profile Disallow: /slots Disallow: /payments Disallow: /static/css/ Disallow: /static/js/ Disallow: /static/fonts/ Disallow: /cdn-cgi/ User-agent: ClaudeBot Allow: / Disallow: /api/ Disallow: /app Disallow: /admin Disallow: /login Disallow: /register Disallow: /forgot-password Disallow: /reset-password Disallow: /verify-email Disallow: /profile Disallow: /slots Disallow: /payments Disallow: /static/css/ Disallow: /static/js/ Disallow: /static/fonts/ Disallow: /cdn-cgi/ User-agent: PerplexityBot Allow: / Disallow: /api/ Disallow: /app Disallow: /admin Disallow: /login Disallow: /register Disallow: /forgot-password Disallow: /reset-password Disallow: /verify-email Disallow: /profile Disallow: /slots Disallow: /payments Disallow: /static/css/ Disallow: /static/js/ Disallow: /static/fonts/ Disallow: /cdn-cgi/ User-agent: Google-Extended Allow: / Disallow: /api/ Disallow: /app Disallow: /admin Disallow: /login Disallow: /register Disallow: /forgot-password Disallow: /reset-password Disallow: /verify-email Disallow: /profile Disallow: /slots Disallow: /payments Disallow: /static/css/ Disallow: /static/js/ Disallow: /static/fonts/ Disallow: /cdn-cgi/ User-agent: Applebot-Extended Allow: / Disallow: /api/ Disallow: /app Disallow: /admin Disallow: /login Disallow: /register Disallow: /forgot-password Disallow: /reset-password Disallow: /verify-email Disallow: /profile Disallow: /slots Disallow: /payments Disallow: /static/css/ Disallow: /static/js/ Disallow: /static/fonts/ Disallow: /cdn-cgi/ User-agent: FacebookBot Allow: / Disallow: /api/ Disallow: /app Disallow: /admin Disallow: /login Disallow: /register Disallow: /forgot-password Disallow: /reset-password Disallow: /verify-email Disallow: /profile Disallow: /slots Disallow: /payments Disallow: /static/css/ Disallow: /static/js/ Disallow: /static/fonts/ Disallow: /cdn-cgi/ User-agent: cohere-ai Allow: / Disallow: /api/ Disallow: /app Disallow: /admin Disallow: /login Disallow: /register Disallow: /forgot-password Disallow: /reset-password Disallow: /verify-email Disallow: /profile Disallow: /slots Disallow: /payments Disallow: /static/css/ Disallow: /static/js/ Disallow: /static/fonts/ Disallow: /cdn-cgi/ # ============================================================================ # Scrapers — Blocked (scrape for datasets/resale, no value back) # ============================================================================ User-agent: CCBot Disallow: / User-agent: Bytespider Disallow: / User-agent: Diffbot Disallow: / User-agent: Omgilibot Disallow: / User-agent: Omgili Disallow: / User-agent: YouBot Disallow: / User-agent: ImagesiftBot Disallow: / User-agent: img2dataset Disallow: / # ============================================================================ # SEO Crawlers — Rate Limited # ============================================================================ User-agent: AhrefsBot Crawl-delay: 10 User-agent: SemrushBot Crawl-delay: 10 User-agent: DotBot Crawl-delay: 10 User-agent: MJ12bot Crawl-delay: 10 Sitemap: https://helperx.app/sitemap.xml