/make

Make your root files.

Below is a worked example for a made-up bakery. Change the answers, or start from a blank form.

01 The site Site address Name for your Book (optional)

02 robots.txt

Open site Crawlers may read it (a live site) Closed site Ask every crawler to stay out (a staging copy)

Paths crawlers can skip (one per line, starting with /)

Listed paths are public in robots.txt itself. Never list a page to hide it: put private pages behind a login.

AI-training crawlers welcome No special rules Ask AI-training crawlers to stay out GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, meta-externalagent

Add a Sitemap line pointing to /sitemap.xml

03 security.txt Where reports go (email, https:// page or tel:+; one per line)

Expires on

Languages

Optional links: policy, encryption key, thanks page

Policy page Encryption key Acknowledgments page

04 sitemap.xml Page addresses (one per line; a date like 2026-10-01 after one is optional)

Up to 10,000 addresses on your plan. A saved set keeps the first 1,000.

OKOpen site: crawlers may read everything not listed below.

NotePrivate paths are printed in robots.txt for anyone to read. Use it to save crawl effort, never to hide something: protect private pages with a login.

NoteAI-training crawlers are ASKED to stay out. Their operators say they follow robots.txt; nothing here can force a crawler that does not.

OK2 contact lines in security.txt.

OKExpires 2027-08-29 is within a year.

Fix2 sitemap lines were left out (listed under the sitemap).

OK4 addresses in sitemap.xml, all on https://www.bakery.example.com.

robots.txt

goes at https://www.bakery.example.com/robots.txt

# robots.txt for www.bakery.example.com
# A public request to crawlers, not a lock.

# AI-training crawlers, asked to stay out:
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: meta-externalagent
Disallow: /

User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /search

Sitemap: https://www.bakery.example.com/sitemap.xml

security.txt

goes at https://www.bakery.example.com/.well-known/security.txt

# security.txt for www.bakery.example.com (RFC 9116). Put it at /.well-known/security.txt
Contact: mailto:security@bakery.example.com
Contact: https://bakery.example.com/report
Expires: 2027-08-29T23:59:59.000Z
Preferred-Languages: en, es
Canonical: https://www.bakery.example.com/.well-known/security.txt
Policy: https://www.bakery.example.com/security-policy

sitemap.xml

goes at https://www.bakery.example.com/sitemap.xml

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url><loc>https://www.bakery.example.com/</loc></url>
  <url><loc>https://www.bakery.example.com/menu</loc><lastmod>2026-10-01</lastmod></url>
  <url><loc>https://www.bakery.example.com/cakes</loc></url>
  <url><loc>https://www.bakery.example.com/visit</loc></url>
</urlset>

2 lines left out

https://www.bakery.example.com/menu listed twice
https://bakery.example.com/old-page not on https://www.bakery.example.com (one sitemap lists one site)