Day 5: Technical SEO – robots.txt, Sitemap, Canonical
✅ Today you will learn
- Control crawling with robots.txt
- Help discovery with an XML sitemap
- Avoid duplicates with canonical tags
- Use noindex, 301 redirects and status codes correctly
- Add hreflang and basic structured data
🏢 In the real world
The institute moved its site from http://www.sharmacomputers.example/tally-course.html to https://sharmacomputers.example/courses/tally/. Now Google sees four versions of the home page, old links show 404 errors, and the Hindi page competes with the English one. Technical SEO fixes all of this in a few lines.
robots.txt – what may be crawled
robots.txt sits at the root (/robots.txt) and tells crawlers which paths not to crawl. It is not a security tool and does not remove a page from Google – a blocked page can still appear if other sites link to it. To keep a page out of results, use noindex and let it be crawled.
User-agent: * Disallow: /admin/ Disallow: /account/ Allow: / Sitemap: https://sharmacomputers.example/sitemap.xml
Disallow: / blocks the whole site. It is often left behind after development – check robots.txt after every launch.
XML sitemap – what should be found
A sitemap lists the URLs you want indexed (only canonical, indexable, 200-status pages). Submit it in Search Console (Day 7) and reference it in robots.txt. Keep lastmod honest – change it only when the content really changes.
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>https://sharmacomputers.example/courses/tally/</loc>
<lastmod>2026-10-20</lastmod>
</url>
<url>
<loc>https://sharmacomputers.example/courses/python/</loc>
<lastmod>2026-09-12</lastmod>
</url>
</urlset>Canonical, noindex and hreflang
| Problem | Fix |
|---|---|
| http://, https://, www and non-www versions of every page | 301-redirect all to one version (below) and set a self-referencing canonical |
| Same page with ?utm_source= or sorting parameters | Canonical to the clean URL |
| Thank-you page, internal search results, login | noindex (and no links in the sitemap) |
| English and Hindi versions of the same page | hreflang en/hi on both pages pointing to each other, plus x-default |
<!-- the preferred URL of this page --> <link rel="canonical" href="https://sharmacomputers.example/courses/tally/"> <!-- English and Hindi versions of the same page --> <link rel="alternate" hreflang="en" href="https://sharmacomputers.example/courses/tally/"> <link rel="alternate" hreflang="hi" href="https://sharmacomputers.example/hi/courses/tally/"> <link rel="alternate" hreflang="x-default" href="https://sharmacomputers.example/courses/tally/"> <!-- keep a page out of Google (e.g. thank-you page) --> <meta name="robots" content="noindex, follow">
Status codes and redirects
| Code | Meaning | SEO use |
|---|---|---|
| 200 | OK | Normal page |
| 301 | Moved permanently | Page moved for good – passes signals to the new URL |
| 302 / 307 | Moved temporarily | Short-term moves only (e.g. a sale page) |
| 404 | Not found | Deleted page with no replacement – fine if it is truly gone; redirect if there is a close replacement |
| 410 | Gone | Permanently removed on purpose |
| 5xx | Server error | Fix fast – repeated errors slow crawling |
# Force HTTPS and one host name (Apache .htaccess)
RewriteEngine On
RewriteCond %{HTTPS} off [OR]
RewriteCond %{HTTP_HOST} ^www\. [NC]
RewriteRule ^ https://sharmacomputers.example%{REQUEST_URI} [L,R=301]
# Old URL moved permanently to the new one
Redirect 301 /tally-course.html /courses/tally/HTTPS and structured data
HTTPS is a must – browsers mark HTTP pages “Not secure” and it is a light ranking signal. Structured data (JSON-LD using schema.org) describes the page in a machine-readable way and can make it eligible for rich results. Add only what is true and visible on the page, and test it with Google’s Rich Results Test.
<script type="application/ld+json">
{
"@context": "https://schema.org",
"@type": "EducationalOrganization",
"name": "Sharma Computer Institute",
"url": "https://sharmacomputers.example/",
"logo": "https://sharmacomputers.example/images/logo.png",
"telephone": "+91-00000-00000",
"address": {
"@type": "PostalAddress",
"streetAddress": "Shop 4, Example Market",
"addressLocality": "Aligarh",
"addressRegion": "Uttar Pradesh",
"postalCode": "202001",
"addressCountry": "IN"
}
}
</script>Google decides which rich result types it shows. For example, FAQ and HowTo rich results are now shown only for very few sites. Check Google Search Central’s list of supported structured-data features before investing time.
Solved examples
Example 1: Diagnose the migration
| Symptom | Cause | Fix |
|---|---|---|
| Old /tally-course.html links show 404 | No redirect after URL change | Redirect 301 /tally-course.html /courses/tally/ |
| Search Console: “Duplicate without user-selected canonical” | http/https and www/non-www all live | Force one host with 301 + self canonical |
| New course pages not indexed after 3 weeks | Not in sitemap and no internal links | Add to sitemap, link from /courses/ and home page |
| Whole site vanished from Google | robots.txt has Disallow: / left from staging | Remove it, then request indexing of key pages |
Practice exercises (unsolved – answer hidden)
Try each exercise yourself first – on paper, in a text editor or on your own site – then open the answer to check.
🏋️ Practice – do it yourself
Show solution
User-agent: * Disallow: /cart/ Disallow: /search/ Allow: / Sitemap: https://sharmacomputers.example/sitemap.xml
Show solution
(a) 301 (b) 302 (temporary) (c) canonical to the main article (d) noindex.Show solution
<link rel="alternate" hreflang="en" href="https://sharmacomputers.example/courses/python/"> <link rel="alternate" hreflang="hi" href="https://sharmacomputers.example/hi/courses/python/"> <link rel="alternate" hreflang="x-default" href="https://sharmacomputers.example/courses/python/">
Put the same set on both the English and the Hindi page.
🏅 Job skills unlocked today
Quick quiz
🧠 Quick Quiz – test yourself
Interview corner
💼 Interview Corner
What is the difference between robots.txt Disallow and noindex?
Disallow stops crawling but the URL can still be indexed from links; noindex lets Google crawl the page and tells it not to show it in results. To de-index, use noindex and do not block the page.
How do you handle a site migration?
Map every old URL to the closest new URL, implement 301 redirects, update internal links and canonicals, keep robots.txt open, submit the new sitemap, and monitor Search Console for 404s and coverage changes.
Summary
- robots.txt controls crawling, not indexing.
- Sitemap lists canonical, indexable URLs.
- One version of every URL: 301 + canonical; hreflang for languages.
- HTTPS always; structured data only for true, visible facts.
Day 6: page speed, mobile-friendliness and Core Web Vitals (LCP, INP, CLS).
