Robots.txt Generator

Builds a robots.txt file that disallows a list of paths for all crawlers. Enter the paths separated by commas, like "/admin, /private".

Builds a robots.txt file that disallows a list of paths for all crawlers. Enter the paths separated by commas, like "/admin, /private".

Use cases

  • Blocking crawlers from indexing an admin or staging area
  • Generating a starting robots.txt for a new site

Examples

Two disallowed paths

/admin, /private
{"content": "User-agent: *\nDisallow: /admin\nDisallow: /private"}

FAQ

Is my data sent to a server?

Yes — the calculation runs on the server for this tool.

Related tools