🚀 SEO, Performance & Deploy · Lesson 28
sitemap.xml, robots.txt & JSON-LD
Why sitemap and robots?
robots.txt tells crawlers what they may visit. sitemap.xml lists the pages you want found, with the date they last changed.
app/sitemap.js
JS – app/sitemap.js
// app/sitemap.js (EduSite – complete file)
import { courses } from '@/lib/courses'
import { getAllPosts } from '@/lib/posts'
import { site } from '@/lib/site'
// Next.js turns this into /sitemap.xml automatically.
export default function sitemap() {
const now = new Date()
const pages = ['', '/courses', '/blog', '/about', '/contact'].map((p) => ({
url: `${site.url}${p}`,
lastModified: now,
changeFrequency: p === '' ? 'weekly' : 'monthly',
priority: p === '' ? 1 : 0.8,
}))
const coursePages = courses.map((c) => ({
url: `${site.url}/courses/${c.slug}`, lastModified: now, priority: 0.9,
}))
const postPages = getAllPosts().map((p) => ({
url: `${site.url}/blog/${p.slug}`, lastModified: new Date(p.date), priority: 0.7,
}))
return [...pages, ...coursePages, ...postPages]
}The generated XML
XML – /sitemap.xml
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>https://edusite.in</loc>
<lastmod>2026-10-19T09:00:00.000Z</lastmod>
<changefreq>weekly</changefreq>
<priority>1</priority>
</url>
<url>
<loc>https://edusite.in/courses/nextjs</loc>
<lastmod>2026-10-19T09:00:00.000Z</lastmod>
<priority>0.9</priority>
</url>
<url>
<loc>https://edusite.in/blog/how-to-start-coding</loc>
<lastmod>2026-03-10T00:00:00.000Z</lastmod>
<priority>0.7</priority>
</url>
…
</urlset>
app/robots.js
JS – app/robots.js
// app/robots.js (EduSite)
import { site } from '@/lib/site'
export default function robots() {
return {
rules: [{ userAgent: '*', allow: '/', disallow: ['/api/'] }],
sitemap: `${site.url}/sitemap.xml`,
}
}TXT – /robots.txt
User-Agent: *
Allow: /
Disallow: /api/
Sitemap: https://edusite.in/sitemap.xml
JSX – noindex a single page
// app/login/page.jsx — keep it out of Google results
export const metadata = { title: 'Login', robots: { index: false, follow: true } }Large sites: multiple sitemaps
One sitemap file can hold at most 50,000 URLs. Use generateSitemaps to split:
JS – generateSitemaps
// app/products/sitemap.js — splits 120,000 products into files of 50,000
export async function generateSitemaps() {
return [{ id: 0 }, { id: 1 }, { id: 2 }]
}
export default async function sitemap({ id }) {
const start = Number(await id) * 50000
const products = await getProducts(start, 50000)
return products.map((p) => ({ url: `https://shop.in/products/${p.slug}`, lastModified: p.updatedAt }))
}
// → /products/sitemap/0.xml, /products/sitemap/1.xml, /products/sitemap/2.xmlSubmit to Google Search Console
- 1Open search.google.com/search-console and add your domain
- 2Verify ownership (DNS TXT record or HTML tag via metadata.verification)
- 3Sitemaps → enter sitemap.xml → Submit
- 4Check "Pages" report after a few days for indexed / not indexed
Helping pages get indexed
- Link every important page from the menu, footer or related posts — orphan pages index slowly.
- Unique title, description and real content (300+ helpful words).
- Correct canonical; no duplicate URLs with
?ref=. - Fast pages (next/image, static rendering) — crawl budget goes further.
- Use URL Inspection → "Request indexing" for brand-new key pages.
Common mistakes
SITE_URLnot set at build → sitemap listshttp://localhost:3000.- Disallowing
/_next/in robots → Google cannot load CSS/JS and sees a broken page. - Listing redirected or 404 URLs → Search Console errors.
Summary
app/sitemap.jsreturns an array of{ url, lastModified }→/sitemap.xml.app/robots.js→/robots.txtwith the sitemap link.- Submit once in Search Console; keep pages linked and canonical.
✅ Practice
Add a fourth Markdown post, rebuild and confirm it appears in /sitemap.xml without touching sitemap.js.
Frequently Asked Questions
Does a sitemap guarantee indexing?
No. It helps Google discover URLs faster, but pages are indexed only if they are useful, unique and linked from other pages.
Should I list every URL in the sitemap?
List only canonical, indexable pages that return 200. Do not include redirects, 404s, login pages or URLs with ?query parameters.
Does robots.txt Disallow remove a page from Google?
No. It only stops crawling. To keep a page out of results, allow crawling and add a noindex robots meta tag.
