Free tutorials & notes in Hindi & English · Clean code examples 🧰 Free PDF & Image Tools →
🚀 SEO, Performance & Deploy  ·  Lesson 28

sitemap.xml, robots.txt & JSON-LD

Why sitemap and robots?

GooglebotYour siteGET /robots.txtAllow /, Sitemap: …/sitemap.xmlGET /sitemap.xmllist of 14 URLs + lastmodcrawl each URLdecide what to index
How Google discovers your pages

robots.txt tells crawlers what they may visit. sitemap.xml lists the pages you want found, with the date they last changed.

app/sitemap.js

JS – app/sitemap.js
// app/sitemap.js  (EduSite – complete file)
import { courses } from '@/lib/courses'
import { getAllPosts } from '@/lib/posts'
import { site } from '@/lib/site'

// Next.js turns this into /sitemap.xml automatically.
export default function sitemap() {
  const now = new Date()
  const pages = ['', '/courses', '/blog', '/about', '/contact'].map((p) => ({
    url: `${site.url}${p}`,
    lastModified: now,
    changeFrequency: p === '' ? 'weekly' : 'monthly',
    priority: p === '' ? 1 : 0.8,
  }))
  const coursePages = courses.map((c) => ({
    url: `${site.url}/courses/${c.slug}`, lastModified: now, priority: 0.9,
  }))
  const postPages = getAllPosts().map((p) => ({
    url: `${site.url}/blog/${p.slug}`, lastModified: new Date(p.date), priority: 0.7,
  }))
  return [...pages, ...coursePages, ...postPages]
}
static pages5 URLscourses[]6 URLscontent/posts/*.md3 URLs/sitemap.xml14 URLs
New course or post = sitemap updates automatically

The generated XML

XML – /sitemap.xml
<?xml version="1.0" encoding="UTF-8"?> <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"> <url> <loc>https://edusite.in</loc> <lastmod>2026-10-19T09:00:00.000Z</lastmod> <changefreq>weekly</changefreq> <priority>1</priority> </url> <url> <loc>https://edusite.in/courses/nextjs</loc> <lastmod>2026-10-19T09:00:00.000Z</lastmod> <priority>0.9</priority> </url> <url> <loc>https://edusite.in/blog/how-to-start-coding</loc> <lastmod>2026-03-10T00:00:00.000Z</lastmod> <priority>0.7</priority> </url> … </urlset>

app/robots.js

JS – app/robots.js
// app/robots.js  (EduSite)
import { site } from '@/lib/site'

export default function robots() {
  return {
    rules: [{ userAgent: '*', allow: '/', disallow: ['/api/'] }],
    sitemap: `${site.url}/sitemap.xml`,
  }
}
TXT – /robots.txt
User-Agent: * Allow: / Disallow: /api/ Sitemap: https://edusite.in/sitemap.xml
JSX – noindex a single page
// app/login/page.jsx — keep it out of Google results
export const metadata = { title: 'Login', robots: { index: false, follow: true } }

Large sites: multiple sitemaps

One sitemap file can hold at most 50,000 URLs. Use generateSitemaps to split:

JS – generateSitemaps
// app/products/sitemap.js — splits 120,000 products into files of 50,000
export async function generateSitemaps() {
  return [{ id: 0 }, { id: 1 }, { id: 2 }]
}

export default async function sitemap({ id }) {
  const start = Number(await id) * 50000
  const products = await getProducts(start, 50000)
  return products.map((p) => ({ url: `https://shop.in/products/${p.slug}`, lastModified: p.updatedAt }))
}
// → /products/sitemap/0.xml, /products/sitemap/1.xml, /products/sitemap/2.xml

Submit to Google Search Console

  1. 1Open search.google.com/search-console and add your domain
  2. 2Verify ownership (DNS TXT record or HTML tag via metadata.verification)
  3. 3Sitemaps → enter sitemap.xml → Submit
  4. 4Check "Pages" report after a few days for indexed / not indexed
One-time setup

Helping pages get indexed

  • Link every important page from the menu, footer or related posts — orphan pages index slowly.
  • Unique title, description and real content (300+ helpful words).
  • Correct canonical; no duplicate URLs with ?ref=.
  • Fast pages (next/image, static rendering) — crawl budget goes further.
  • Use URL Inspection → "Request indexing" for brand-new key pages.

Common mistakes

  • SITE_URL not set at build → sitemap lists http://localhost:3000.
  • Disallowing /_next/ in robots → Google cannot load CSS/JS and sees a broken page.
  • Listing redirected or 404 URLs → Search Console errors.

Summary

  • app/sitemap.js returns an array of { url, lastModified } → /sitemap.xml.
  • app/robots.js → /robots.txt with the sitemap link.
  • Submit once in Search Console; keep pages linked and canonical.
✅ Practice

Add a fourth Markdown post, rebuild and confirm it appears in /sitemap.xml without touching sitemap.js.

Frequently Asked Questions

Does a sitemap guarantee indexing?
No. It helps Google discover URLs faster, but pages are indexed only if they are useful, unique and linked from other pages.
Should I list every URL in the sitemap?
List only canonical, indexable pages that return 200. Do not include redirects, 404s, login pages or URLs with ?query parameters.
Does robots.txt Disallow remove a page from Google?
No. It only stops crawling. To keep a page out of results, allow crawling and add a noindex robots meta tag.
← Next.js Tutorial
🔗

Share this topic with a friend

यह topic किसी दोस्त को भेजें

Found it useful? Send it to a classmate learning the same thing.

अच्छा लगा? जो दोस्त यही सीख रहा है, उसे भेज दीजिए।

Next topicPerformance & Core Web Vitals Your progress: 28 / 37 (76%) · Lesson 29 / 37 Continue →

How was this lesson?

Your feedback helps us make the content better.

Write a review or suggestion (optional)

After approval, your name and city may appear publicly with your review. Please do not write your email or phone number.

▶ CodingEasily on YouTube

Learn with video lessons in easy Hindi

Featured video: RDBMS Part 3 – Class XI Commerce IT

▶ Subscribe – it’s free