SEO Architecture for Web Apps: What Crawlers Need From Your Rendering

What you'll learn: By the end of this lesson you will be able to describe how Google crawls, renders and indexes a JavaScript app, choose which pages need their content in the first HTML, return honest status codes (and avoid soft 404s), set canonical URLs, links and sitemaps correctly, tell robots.txt from noindex, add structured data that matches the page, and check all of it in the built output instead of in a browser.

Picture a librarian who builds a catalogue from the cards in a drawer. If each card has the title and summary, the job is quick. If each card says "see the attached instructions", she may follow them later, and some librarians will not follow them at all. A search crawler is that librarian, and your architecture decides what is written on the card.

This lesson is about what can be fetched, understood and indexed. It is not about keywords or ranking, and it promises nothing about either. It assumes Rendering Strategies: CSR, SSR, SSG, ISR and Streaming. The facts about Google come from Google Search Central documentation, cited with page dates; other search engines differ and are not covered.

The running example is an online shop with category pages, product pages and an account area.

The core idea

1. A crawler needs a URL it may fetch, a response that says what the page is, and content it can read. Rendering, status codes and links are where architecture decides this.
2. Every control is a signal with its own strength and job. A redirect, a canonical link, a sitemap entry, robots.txt and noindex each do one thing; using the wrong one is the usual mistake.
3. Check the output, not the intention. What matters is the HTML and headers a crawler receives, which curl and a script can inspect.

How Google processes a page

Google's JavaScript SEO basics (last updated 2026-03-04) describe three phases: crawling, rendering and indexing. Googlebot takes a URL from its crawl queue, reads robots.txt, and if the URL is allowed fetches it and queues the links in the HTML. It "queues all pages with a 200 HTTP status code for rendering, unless a robots meta tag or header tells Google not to index the page", where the wait is "a few seconds, but it can take longer". A headless Chromium then runs the JavaScript, and the rendered HTML is parsed for links again and used for indexing.

Three details matter. Google "won't render JavaScript from blocked files or on blocked pages". If the status is not 200, "rendering might be skipped". And: "server-side or pre-rendering is still a great idea because it makes your website faster for users and crawlers, and not all bots can run JavaScript."

Figure 1. From URL to index

How to read this: the HTML you send is read twice, as fetched and after rendering. Content already in the first HTML does not wait for the render queue.

Put the content in the first HTML

For search the question is what a crawler gets from the first request, which the rendering strategy decides.

Some teams used dynamic rendering, serving crawlers a prerendered version. Google's documentation (last updated 2025-12-10) calls it "a workaround and not a recommended solution" and recommends "server-side rendering, static rendering, or hydration".

Illustration 1: the minimal example. The first HTML of a product page, written for a crawler:

<head>
  <title>Trail Runner shoe | Example Shop</title>
  <meta name="description" content="A light trail shoe with a grippy sole.">
  <link rel="canonical" href="https://shop.example/products/trail-runner">
</head>
<body>
  <h1>Trail Runner shoe</h1>
  <a href="/category/trail-shoes">More trail shoes</a>
</body>

The title, canonical URL, heading and a real link are all in the HTML as sent.

Status codes tell the truth about a page

Google's HTTP status documentation (last updated 2026-02-04) says how each class is handled:

A soft 404 is a page that says "not found" in the body but answers 200. It is the characteristic bug of apps that route on the client. Google's JavaScript guidance says that in client-rendered apps meaningful status codes "can be impossible or impractical", and offers two ways out: redirect with JavaScript to a URL for which the server answers 404 (for example /not-found), or add <meta name="robots" content="noindex"> to error pages with JavaScript. It warns that Google "may skip rendering and JavaScript execution" when it sees noindex, so removing it later with JavaScript may not work.

The better answer is architectural: decide the status on the server, before any HTML is sent. A streamed page can do that only before streaming starts, because the status is committed with the first chunk (the trap described in the streaming lesson).

Illustration 2: the realistic example. The shop's /products/trail-runner exists. /products/does-not-exist must answer 404 from the server with a "not found" page, and no catch-all may serve the home page for unknown paths.

Canonical URLs and duplicates

One page often has several URLs: tracking parameters, sort orders, www and bare domain, trailing slashes. Google's canonicalization guidance (last updated 2026-07-10) lists ways to say which is preferred, by strength: redirects are a strong signal, rel="canonical" link annotations are a strong signal, and a sitemap entry is a weak signal. A rel="canonical" HTTP header is an option when you control server configuration, including for non-HTML files such as PDFs. The rules worth keeping:

• use absolute URLs in rel="canonical";
• put a self-referencing canonical on the canonical page itself;
• do not give different canonicals through different techniques for the same page;
• do not use robots.txt for canonicalization;
• set it in the HTML: "The best way to set the canonical URL is to use HTML". If JavaScript must set it, it must never change a canonical already in the HTML, and there must be only one canonical link tag.

Links, routing and sitemaps

Google "can only discover your links if they are <a> HTML elements with an href attribute". A click handler on a <div> is invisible. For client-side routing use the History API and real URLs, not fragments (#/products), because "don't use fragments to load different page content".

A sitemap lists the URLs you want in results: "the canonical URLs". Google's sitemap documentation (last updated 2026-07-08) says it is a hint, not a guarantee; one file is limited to 50 MB uncompressed or 50,000 URLs; URLs must be absolute; <priority> and <changefreq> are ignored; and <lastmod> is used only when consistently and verifiably accurate. robots.txt can point to it:

Sitemap: https://shop.example/sitemap.xml

Generate the sitemap from the same data that generates the pages.

robots.txt is not noindex

Google's robots.txt documentation (last updated 2025-12-10) says the file "is used primarily to manage crawler traffic" and "is not a mechanism for keeping a web page out of Google": a blocked URL can still be indexed without its content if other pages link to it. Keeping a page out of results needs noindex (in a robots meta tag or an X-Robots-Tag header), and Google can only read it if crawling is allowed (robots meta tag documentation, last updated 2026-03-24). Blocking a page in robots.txt therefore hides its noindex from Google.

Figure 2. Which control for which problem

How to read this: each problem has one control. The common mistakes are robots.txt used to hide a page and a 200 used to say "gone".

Structured data

Structured data is a standard format for describing a page's content. Google supports JSON-LD, Microdata and RDFa and recommends JSON-LD "if your site's setup allows it" (structured data introduction, last updated 2025-12-10). Two rules matter for architecture. The markup must describe content visible on the page: "don't add structured data about information that is not visible to the user". And markup can make a page eligible for enhanced results, which is not the same as getting them. The JavaScript guidance says JSON-LD may be injected with JavaScript, with testing; rendering it in the HTML is simpler. The properties Google needs for a given rich result are on that feature's own page, which this lesson did not open, so this sketch shows the format only:

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "Product",
  "name": "Trail Runner shoe",
  "description": "A light trail shoe with a grippy sole."
}
</script>

This site as a worked example

This course's own site (buildingai.in) applies these rules in its build, as its project documentation describes. It is one example, not a template.

Figure 3. How the site answers a URL

How to read this: the host decides the status before any JavaScript runs. A catch-all rewrite would remove the lower path.

Trade-offs

In practice: a shop moves its catalogue off client-side rendering

The situation

A shop's category and product pages are a client-rendered app: the first HTML is a shell and a script tag. The team wants the catalogue findable. Unknown product URLs show "Product not found" with a 200.

Starting architecture

Figure 4. Starting architecture: the server answers every URL with the shell

How to read this: nothing in the first response says what the page is, and a missing product looks like a real one until JavaScript runs.

Options considered

The decision

Rollout and verification

1. Fetch like a crawler. curl -s ten sample URLs and check that title, canonical, h1, links and text are present without JavaScript.
2. Check statuses. A missing product and a typo URL must return 404 from the server; an old URL that moved must return 301.
3. Lint the built output in CI: one canonical per page, every sitemap URL answers 200, no internal link to a 404.
4. Submit the sitemap and watch Search Console's indexing and soft-404 reports for a few weeks.

The 2 a.m. failure

A routing change adds a catch-all rewrite "so deep links always work". Pages load and tests pass. Two weeks later Search Console lists soft 404 errors for URLs nobody linked to. The catch-all returns the home page with a 200, so each unknown URL looks like a duplicate home page with a success status.

First three checks: (1) curl -I a nonsense URL and read the status line; (2) read the host's rewrite rules for a pattern that matches everything; (3) compare the soft-404 URLs with the rewrite's scope. Mitigation: remove the catch-all and list the client-only routes explicitly. Follow-up: a deploy check that requests a random path and fails unless it gets a 404.

Architect's notes

• Decide per page type what must be in the first HTML; the rest can be client-rendered.
• Never use 200 for "not found".
• One URL per page. Fix duplicates with redirects first, canonicals second.
• Run the lint in CI, not once at launch: regressions in the output are quiet.
• Fingerprint your assets. Google's renderer may ignore caching headers, so a file name that changes with its content avoids stale scripts.

Common mistakes

• Hiding pages with robots.txt. A blocked URL can still be indexed, and its noindex is never read.
• Using JavaScript to remove noindex. Google may not render a page that has it.
• Navigating with click handlers and fragments instead of <a href>.
• A catch-all rewrite that returns 200.
• Structured data that describes content the page does not show.

Key takeaways

• Google crawls, renders (when the status is 200) and indexes; content in the first HTML does not wait for rendering.
• Prefer server-side, static or incremental rendering for pages that must be found; Google calls dynamic rendering a workaround.
• Return the true status: a 200 page saying "not found" is a soft 404.
• Redirects and rel="canonical" are strong signals, sitemaps a weak one; set canonicals in HTML.
• Links must be <a href>; routes must be real URLs.
• robots.txt manages crawling; noindex manages indexing and needs crawling to be read.
• A sitemap is a hint and should list canonical URLs only.
• Structured data must match visible content; it makes a page eligible for enhanced results, not entitled to them.
• Verify the built output with a script on every change.

Practice exercises

Exercise 1 - Starter (30 min): curl -s three pages of a site you know. List the text, links, title and canonical in the HTML before JavaScript runs.

Exercise 2 - Intermediate (1 hour): Request a nonsense path on that site with curl -I. Write what you got, what it should have been, and which layer decides.

Exercise 3 - Advanced (half day): Write a small script that reads a built site's sitemap and checks each URL for status 200, one canonical equal to the URL, and one h1. Add a request for a random path that must return 404.

Related reading

In this course:

• Rendering Strategies: CSR, SSR, SSG, ISR and Streaming
• Streaming SSR and Suspense, why the status is committed at the first byte.
• Incremental Static Regeneration and On-Demand Revalidation
• HTTP Caching, fingerprinted assets.

Official documentation

• Google: Understand JavaScript SEO Basics (updated 2026-03-04), crawling, rendering, soft 404s in single-page apps.
• Google: How HTTP Status Codes Affect Google's Crawlers (updated 2026-02-04), how each status is handled.
• Google: How to Specify a Canonical with rel="canonical" and Other Methods (updated 2026-07-10), canonical signals.
• Google: Build and Submit a Sitemap (updated 2026-07-08).
• Google: Robots Meta Tags Specifications (updated 2026-03-24) and Robots.txt Introduction and Guide (updated 2025-12-10).
• Google: Intro to How Structured Data Markup Works and Dynamic Rendering as a workaround, both updated 2025-12-10.