Robots.txt Generator

Generate robots.txt file for your website

Free to sign up

Growth & Brand Lab SEO Tools 🟢 Легко

Preview mode. Complete the step above to save, export, or generate results.

Website URL

Preset Template

Block AI Crawlers (GPTBot, CCBot, etc.)

Block Ad Crawlers (AdsBot)

Block SEO Tool Bots (SemrushBot, AhrefsBot)

Crawl-delay (seconds, 0 = none)

Include Sitemap reference

Sitemap URL

Placement: Upload to your root domain: https://yourdomain.com/robots.txt

Case Sensitive: Paths are case-sensitive. /Admin/ ≠ /admin/

Not Security: robots.txt is advisory only. Don't rely on it to hide sensitive content — use authentication instead.

Wildcards: Use * for pattern matching. Example: Disallow: /*.pdf$ blocks all PDFs.

Testing: Use Google Search Console to test your robots.txt.

Our team can help you with personalized guidance for your startup.

Necessary

Essential for the site to function. Cannot be disabled.

Always On

Analytics

Help us understand how visitors interact with the site (Google Analytics, Yandex Metrika).

Marketing

Used to show relevant promotions and offers based on your activity.

Preferences

Remember your settings such as language and theme.