SEO Tools
Build a robots.txt without breaking your site
Pick your crawlers, add your rules, and copy the file. The one mistake that matters — a stray Disallow: / that removes your whole site from search — is called out loudly, because it looks almost identical to the harmless version that allows everything.
Start from
Sitemap URL
robots.txt
Save this as robots.txt in the root of your site, so it is served at yoursite.com/robots.txt. It only works from the root — a copy in a subfolder is ignored.
- Runs in your browser
- No account required
- Free to use
How it works
Groups, then rules
Each block names a user-agent and lists what it may and may not fetch. Crawlers obey the first block that matches their name, which is why a duplicated user-agent silently does nothing — and why this tool tells you when you have one.
Disallow: / and Disallow: are opposites
An empty Disallow allows everything. A Disallow with a single slash blocks everything. They differ by one character and read almost the same, so the destructive one is flagged as an error rather than a note.
The sitemap line is global
Sitemap is not part of any user-agent group, so it is written at the end of the file. Putting it inside a group is a common mistake that makes it look conditional when it is not.
What it does
Every crawler that matters
Google, Bing, DuckDuckGo and Yandex, plus GPTBot, ClaudeBot, PerplexityBot, Google-Extended and CCBot — because whether to allow AI crawlers is a real decision right now.
Presets to start from
Allow everything, block AI training crawlers, or a WordPress starting point that keeps wp-admin out of the index without breaking admin-ajax.
Validation as you type
Missing slashes, spaces in paths, duplicate user-agents, relative sitemap URLs and non-numeric crawl-delays — caught before you publish rather than after.
It tells you what robots.txt cannot do
Blocking a path here does not hide it. The file is public, so listing a private directory advertises it. The tool says so when it spots one.
Practical advice
Getting a better result
- robots.txt controls crawling, not indexing. A blocked page can still appear in results if other sites link to it — use a noindex meta tag to keep a page out of the index.
- Anyone can read your robots.txt. Listing /admin/ or /backup/ tells people exactly where to look; use real authentication for anything that matters.
- Blocking your CSS or JavaScript stops Google rendering your pages the way visitors see them, which can hurt more than the crawl budget it saves.
- Google ignores Crawl-delay entirely. If Googlebot is hitting you too hard, set the crawl rate in Search Console instead.
Questions
Things people ask
Where does the file go?
In the root of your domain, served at yoursite.com/robots.txt. It has to be at the root — a copy in a subfolder is ignored completely.
On most hosts that means the same folder as your homepage. On a framework or CMS there is usually a public or static directory that maps to the root.
Does robots.txt keep pages out of Google?
Not reliably. It asks crawlers not to fetch a page, but a page that is linked from elsewhere can still be indexed — sometimes with no description, because the crawler was not allowed to read it.
To keep something out of the index, use a noindex meta tag on the page and let crawlers reach it. To keep something private, use authentication.
Is it safe to block AI crawlers?
Yes, in the sense that GPTBot, ClaudeBot and Google-Extended are separate from the crawlers that index you for search. Blocking them does not affect your Google or Bing ranking.
Whether it is a good idea is a business decision rather than a technical one, which is why the tool offers it as a preset rather than a recommendation.
Why is my rule being ignored?
The most common reason is a duplicate user-agent: crawlers follow only the first block that matches their name, so a second block for the same agent does nothing. This tool flags that.
The other common reason is a missing leading slash. Paths are matched from the root, so admin/ is not the same as /admin/.
Do I even need a robots.txt?
If you want everything crawled, no — a missing robots.txt means the same thing as one that allows everything. It becomes useful when you have parts of the site that are not worth crawling, or when you want to point crawlers at your sitemap.
Related tools
Website SEO Analyzer
SoonCheck a page for on-page, technical, social and accessibility SEO.
Fetches the URL you enter
Meta Tag Generator
Build title, description, Open Graph and Twitter tags with a live preview.
Runs entirely in your browser
Open toolSitemap Generator
Turn a list of URLs into a valid XML sitemap, with the mistakes flagged.
Runs entirely in your browser
Open toolSchema Markup Generator
Build JSON-LD for nine types, checked against what Google actually accepts.
Runs entirely in your browser
Open toolWho builds these
Free tools by day. Websites that earn their keep by trade.
KuConvert is built by KUDUY, a web design and development agency. From custom WordPress builds to SEO and e-commerce, everything they do is engineered to bring you more qualified traffic and turn it into revenue.
Web Design
Custom, conversion-focused web design that captivates visitors and turns them into paying customers. Mobile-first, fast, and built to rank.
Web Development
Custom WordPress websites built with clean code, blazing-fast performance (Core Web Vitals optimised), and enterprise-grade security.
E-Commerce
Online stores that convert visitors to customers with seamless shopping experiences.
SEO
Technical SEO, keyword research, on-page optimisation and local SEO strategies that drive qualified organic traffic.
Digital Marketing
Data-driven campaigns that deliver measurable ROI across every channel.
Branding
Memorable brand identities that stand out in any market and build lasting trust.
Website Help
One-on-one expert support for bug fixes, updates and ongoing maintenance — without a long-term retainer.
Fixed scope, no surprise invoices
Most sites ship in four to eight weeks with the scope and the price agreed up front.
SEO from day one
Search is designed into the build — structure, schema and Core Web Vitals — not retrofitted.
A team, not a ticket queue
A project manager, a designer and a developer you can actually reach.
Not sure what should be crawlable?
Crawl budget, index bloat and duplicate content are the kind of thing that quietly holds a site back. KUDUY does technical SEO audits that find them.
This tool runs entirely in your browser. Your files are never uploaded.