Build a robots.txt with mistake warnings, or an XML sitemap with image & video support.
robots.txt and sitemap.xml serve opposite purposes — one restricts what crawlers can access, the other actively points them toward everything you want found — and both are easy to get subtly wrong by hand. This tool builds either one, starting from CMS presets for robots.txt and warning you directly if a rule would accidentally block your entire site, plus image and video sitemap support for content beyond plain pages.
At the root of your domain — for example, kravoy.com/robots.txt. It must be at that exact top-level location for search engines to find it; it won't work in a subfolder.
Disallow tells a crawler not to access a path. Allow explicitly permits a path, which is mainly useful for carving out an exception inside a broader Disallow rule — for example, blocking a whole plugin folder except one specific file crawlers need.
It should include every page you want indexed and discoverable, but doesn't need to include pages you've deliberately excluded from search (like admin pages or duplicate content), which is worth aligning with your robots.txt rules.
They're hints, not commands — changefreq suggests how often a page's content changes, and priority suggests its relative importance compared to other pages on the same site. Search engines treat both as optional signals, not strict rules they must follow.
Yes, by pasting your full URL list — for sites large enough to need multiple sitemap files (typically beyond 50,000 URLs per file), you'd split the list and generate each file separately, then reference all of them from a sitemap index file.
Yes, completely free, with no sign-up and no limit on file size beyond what your browser can comfortably handle.
