0Pricing
HTML Academy · Lesson

Canonical URLs and robots meta

Control crawling and avoid duplicate content with meta tags.

Canonical URLs and robots meta is a free HTML Academy lesson on CoddyKit — lesson 3 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the HTML Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Duplicate Content Problem

Search engines may encounter your page at multiple URLs:

  • http://example.com/page
  • https://example.com/page
  • https://www.example.com/page
  • https://example.com/page/ (trailing slash)
  • https://example.com/page?utm_source=newsletter

Google must decide which URL to index — and may penalize for duplicate content.

The canonical Link Tag

Tell search engines the preferred URL with rel="canonical":

<head>
  <!-- This is the canonical URL for this content: -->
  <link rel="canonical" href="https://example.com/blog/html5-guide">
</head>
<!-- No matter which URL the page is accessed from,
     this tag signals to Google which URL to index -->
<!-- Link equity is consolidated at the canonical URL -->

Canonical URL Rules

Canonical URL best practices:

  • Use the full, absolute URL (including protocol)
  • Use HTTPS if your site supports it
  • Be consistent: always with or without trailing slash
  • Every page should have a canonical tag — even if it's its own URL
<link rel="canonical" href="https://example.com/blog/html5-guide">
<!-- NOT: -->
<!-- <link rel="canonical" href="/blog/html5-guide"> relative = bad -->
<!-- <link rel="canonical" href="http://..."> HTTP on HTTPS site = bad -->

Pagination and canonical

For paginated content, canonical may point to the first page or itself:

<!-- Page 2 of a blog listing: -->
<link rel="canonical" href="https://example.com/blog?page=2">
<!-- Or: consolidate all pages to the first -->
<link rel="canonical" href="https://example.com/blog">
<!-- Also useful: rel="prev" and rel="next" for paginated series -->

meta robots Values

Control search engine behavior per page:

<meta name="robots" content="index, follow">
<!-- Default: include in index and follow links -->

<meta name="robots" content="noindex">
<!-- Don't index this page -->

<meta name="robots" content="nofollow">
<!-- Don't follow links on this page -->

<meta name="robots" content="noindex, nofollow">
<!-- Don't index AND don't follow -->

<meta name="robots" content="noarchive">
<!-- Don't show cached version in search results -->

Googlebot-Specific Directives

Target specific crawlers with specific directives:

<meta name="googlebot" content="noindex">
<!-- Only affects Google's crawler -->

<meta name="bingbot" content="noindex">
<!-- Only affects Bing's crawler -->

<!-- Other values: -->
<meta name="robots" content="max-snippet:50">
<!-- Limit snippet length in search results to 50 characters -->

<meta name="robots" content="max-image-preview:large">
<!-- Allow full-size image previews in Google results -->

X-Robots-Tag HTTP Header

Canonical and robots can also be set via HTTP headers (server-level):

# Apache .htaccess:
Header add Link "<https://example.com/page>; rel=canonical"
Header add X-Robots-Tag "noindex"

# This is more powerful because it applies to non-HTML files too
# (PDFs, images, etc.) which can't have meta tags

robots.txt vs meta robots

Two tools for controlling crawlers — different purposes:

  • robots.txt — controls which pages crawlers can access (access control)
  • meta robots — controls what crawlers do with pages they've accessed (indexing control)

A page disallowed in robots.txt may still be indexed if linked from elsewhere. A page with noindex will definitely not be indexed.

Sitemap and Canonical

Your sitemap should only list canonical URLs:

<!-- sitemap.xml should list canonical URLs only: -->
<!-- https://example.com/blog/html5-guide (canonical) -->
<!-- NOT: https://www.example.com/blog/html5-guide -->
<!-- NOT: https://example.com/blog/html5-guide/ (trailing slash variant) -->

Self-Referential Canonical

Every page should have a self-referential canonical — even the main one:

<!-- On https://example.com/about (the canonical URL itself): -->
<link rel="canonical" href="https://example.com/about">
<!-- This confirms: this IS the canonical URL -->
<!-- Prevents issues if the page is embedded in an iframe or scraped -->

Canonical in E-commerce

E-commerce canonical patterns for products with variants:

<!-- Product page with color filter: -->
<!-- URL: /products/widget?color=blue -->
<link rel="canonical" href="https://example.com/products/widget">
<!-- All color variants point to the main product page -->
<!-- Avoids duplicate content penalty for parameter variants -->

Quick Check

What should the href value of a canonical link tag be?

Recap: Canonical and robots

Canonical and robots essentials:

  • <link rel="canonical" href="full-url"> — declare the preferred URL
  • Use absolute URLs in canonical; always HTTPS
  • meta robots noindex — exclude from search index
  • meta robots nofollow — don't follow links
  • robots.txt = access control; meta robots = indexing control

Frequently asked questions

Is the “Canonical URLs and robots meta” lesson free?

Yes — the full text of “Canonical URLs and robots meta” is free to read here on the web, and the HTML Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the HTML Academy course, upgrade to CoddyKit PRO.

What will I learn in “Canonical URLs and robots meta”?

Control crawling and avoid duplicate content with meta tags. You practise HTML Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start HTML Academy?

No prior experience is required. HTML Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 3 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Canonical URLs and robots meta” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this HTML Academy lesson?

Yes. Every HTML Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

  1. title and its Role in SEO
  2. meta charset viewport author and description
  3. Canonical URLs and robots meta
  4. Favicon and Apple Touch Icon
← Back to HTML Academy