A properly configured sitemap and robots.txt help search engines discover and correctly index your static site's content — this guide covers generating and maintaining both.
Why These Files Matter for a Static Site
Unlike a CMS that might generate these dynamically, a static site typically needs these generated as part of your build process, requiring deliberate setup — without them, search engines rely purely on link discovery, which can be less efficient and complete.
Generating a Sitemap Automatically at Build Time
npm install sitemapconst { SitemapStream, streamToPromise } = require('sitemap');
const sitemap = new SitemapStream({ hostname: 'https://yourdomain.com' });
pages.forEach(page => sitemap.write({ url: page.path, changefreq: 'weekly' }));
sitemap.end();
streamToPromise(sitemap).then(data => fs.writeFileSync('dist/sitemap.xml', data));Generating the sitemap programmatically from your actual page list ensures it stays automatically synchronized with your content, rather than requiring manual maintenance.
Many Static Site Generators Have Built-In Sitemap Plugins
Astro, Next.js, Hugo, and most popular static site generators have official or well-maintained community plugins for automatic sitemap generation — prefer these over a fully custom implementation where available, since they handle framework-specific routing nuances correctly.
Basic Sitemap Structure
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>https://yourdomain.com/</loc>
<lastmod>2026-08-15</lastmod>
<changefreq>weekly</changefreq>
</url>
</urlset>Creating robots.txt
User-agent: *
Allow: /
Sitemap: https://yourdomain.com/sitemap.xmlA simple robots.txt referencing your sitemap location — for most static sites, allowing all crawling with a sitemap reference is the appropriate default configuration.
Excluding Specific Paths (If Needed)
User-agent: *
Disallow: /admin/
Disallow: /draft/
Allow: /If your static site includes paths that shouldn't be indexed (draft content, internal tooling pages), explicitly disallow them — though be aware robots.txt is advisory, not a genuine access control mechanism; sensitive content needs actual access restriction, not just a disallow directive.
Placing Both Files Correctly
Both sitemap.xml and robots.txt must be served from your domain root (https://yourdomain.com/sitemap.xml, https://yourdomain.com/robots.txt) — ensure your build output places them correctly for your Nginx configuration to serve them from the root path.
Submitting Your Sitemap to Search Engines
Submit your sitemap URL through Google Search Console and equivalent tools for other search engines — accelerates discovery, particularly valuable for new sites or significant content additions.
Keeping the Sitemap Current
Since it's regenerated on every build (if properly automated), your sitemap should automatically reflect content changes — verify this is genuinely happening by checking the generated sitemap after a content update, rather than assuming automation is working correctly without confirmation.
Common Errors
Sitemap references pages that return 404 — usually indicates stale generation logic not accounting for removed content; ensure your sitemap generation genuinely reflects your current build output, not a cached or manually-maintained list.
Continue Reading
- How to Configure Nginx for Single Page Applications (SPA Routing)
- Understanding Core Web Vitals for Static Sites
- How to Handle 404 Pages Correctly on a Static Site
Browse more articles in Static Site Hosting & Frontend Deployment.
