How to create a robots.txt file

A robots.txt file gives crawlers basic instructions about which paths may be crawled. For many small websites, a simple file that allows crawling and points to the sitemap is enough.

A basic robots.txt file can include "User-agent: *", "Allow: /", and a Sitemap line. If you need to block private paths, add Disallow rules carefully. Use the Robots.txt Generator to create a clean starting point.

Do not use robots.txt to protect sensitive information. Blocked URLs can still be discovered, and private content should be protected with proper access controls.