Root Files Pass $1.95 a month or $19.50 a year Free during preview

What it adds

/guide

Where the files go, and what the checks mean.

All three files live at the top of your website, so a browser can open them at a fixed address.

  1. https://your-site/robots.txt

    In the same folder as your home page (often public_html or www on shared hosting). One file per host: www. and the bare name each need their own if both serve pages.

  2. https://your-site/sitemap.xml

    Same folder. The robots.txt Sitemap line points crawlers to it, and you can also add it in the search engines' webmaster tools.

  3. https://your-site/.well-known/security.txt

    In a folder named .well-known (the dot matters) inside that same folder. Many hosts already have that folder for HTTPS certificates; add the file beside what is there.

On WordPress and other systems that write their own robots.txt or sitemap, use the system's setting or plugin for that file instead of uploading a second one. After uploading, open each address in a browser: it should show plain text (or XML), not your home page.

The checks

C1 Closed site
"Disallow: /" for every crawler. Right for a staging copy; on a live site it asks search engines to stop reading every page.
C3 Listing is not hiding
robots.txt is public. Paths in it are visible to anyone. It is a request to well-behaved crawlers, not a lock.
C4 Expires
RFC 9116 requires an Expires field and recommends a date less than a year away. A file past its date should be treated as stale.
C5 Contact
RFC 9116 requires at least one Contact line: an email (written as mailto:), an https:// page or a phone number.
C6 One site per sitemap
The sitemaps.org protocol: all addresses in one sitemap are on the same host as the sitemap. Others are left out and listed, as are duplicates and lines that are not full addresses.
C8 Size
A sitemap may hold up to 50,000 addresses by the protocol. Root Files writes up to 500 free, 10,000 with the Pass; past that, split the list.
C9 Empty sitemap
robots.txt points to a sitemap with no addresses yet.

The AI-training crawlers

When you ask them to stay out, robots.txt names these crawler tokens, as their operators publish them:

Operators add and rename crawlers. This list is reviewed when the site is updated; it is not a promise that every AI crawler is named, or that any crawler obeys.