Technical SEO is the work that makes a site crawlable, indexable, and usable, independent of content quality. Even the best content can’t compete in any query if it’s sitting behind a technical blocker. That’s why the technical side is SEO’s invisible but foundational layer.
The good news: for most sites, technical SEO isn’t an area that demands constant effort — it’s infrastructure you set up correctly once and audit periodically. The bad news: a single wrong line (say, a Disallow in robots.txt) can knock the entire site out of results.
The checks in this article rest on these official sources: the robots.txt documentation defines crawling rules, the sitemap documentation defines the discovery file, the canonical documentation defines duplicate URL management, and web.dev’s Core Web Vitals guide defines the speed metrics.
Technical SEO Audit Checklist
Not every issue in a technical audit carries the same priority. Issues that block crawling and indexing get fixed before speed or schema improvements.
| Check | Risk | Priority |
|---|---|---|
| HTTPS usage | Trust warnings, weaker user experience, and signal loss. | High |
| Site versions | Signal splitting across http, https, www, and non-www versions. | High |
| robots.txt | Important directories accidentally closed off to crawling. | High |
| Crawl errors | 404s, 5xx errors, redirect loops, or robots blocks. | High |
| Sitemap | Important URLs missing from the discovery list. | Medium |
| Canonical | The wrong official address chosen among duplicate URLs. | High |
| Broken links | Disrupted bot and user flow. | Medium |
| HTTP resource on an HTTPS page | Mixed content and trust issues. | Medium |
| URL structure | Meaningless, parameter-heavy, or constantly changing addresses. | Medium |
| Page depth | An important page sitting too many clicks from the homepage. | Medium |
| Redirects | Using 302 for a permanent move, or redirect chains. | Medium |
| Structured data | The page’s type and related information not fully communicated to machines. | Low-Medium |
| Core Web Vitals | A user-experience disadvantage among otherwise similar-quality results. | Medium |
Use this table alongside crawler output, the Search Console index report, and URL Inspection results. No single tool is enough on its own: the crawler shows the structure the site presents, while Search Console shows what Google actually sees.
The Crawling Layer: Can Googlebot Reach the Site?
robots.txt: the boundary drawn for crawlers
This plain-text file sitting at the site’s root tells crawlers which sections they can enter. Blocking off the admin panel or filter parameters is a legitimate use; the risk is a rule written too broadly that also blocks sections you actually need. A Disallow: / line on its own blocks the entire site from crawling. A line like that left over from a staging environment when a site goes live is the classic technical SEO accident.
One more caveat: robots.txt is not a de-indexing tool. A blocked page can keep showing up in the index if it has links from elsewhere; the correct way to remove a page from the index is a noindex tag.
AI bots: GPTBot, OAI-SearchBot, and CCBot are not the same thing
When AI search visibility comes up, the phrase “block GPTBot” gets used a lot — but these bots don’t all do the same job. According to OpenAI’s crawler documentation, OAI-SearchBot is used to surface websites in ChatGPT’s search results. OpenAI recommends allowing OAI-SearchBot access in robots.txt if you want ChatGPT Search visibility. GPTBot, on the other hand, is used to crawl content that may be used to train generative foundation models; blocking it signals that your content shouldn’t be used for model training. ChatGPT-User isn’t an automated web crawl at all — it’s used for certain user-initiated ChatGPT actions; OpenAI states that search visibility decisions shouldn’t be managed through this agent.
On the Common Crawl side, CCBot is a separate bot that produces web crawl data. Common Crawl explains that CCBot checks robots.txt first, that its user-agent value is CCBot, and that User-agent: CCBot / Disallow: / can be used to block it.
In practice, this distinction means:
| Bot | Core purpose | robots.txt decision |
|---|---|---|
OAI-SearchBot | Surfacing websites in ChatGPT search results | Shouldn’t be blocked if you want ChatGPT Search visibility. |
GPTBot | Crawling for OpenAI foundation model training | Can be separately blocked if you don’t want your content used for model training. |
ChatGPT-User | User-initiated ChatGPT visits | Search visibility decisions aren’t made through this agent. |
CCBot | Web crawling for the Common Crawl dataset | Allowed or blocked depending on your open-data/crawl strategy. |
In short, “I want to show up in AI searches” and “I want to contribute data to model training” are not the same decision. When reviewing your robots.txt file, think of these AI bots as separate lines alongside Googlebot and Bingbot — and don’t wave off the commercial, legal, and content-licensing dimension as just another routine technical SEO item.
The sitemap: an invitation list for discovery
sitemap.xml is the file listing the URLs you want indexed. It’s an invitation, not a guarantee — but it noticeably speeds up discovery, especially for sites that are new, large, or have weak internal linking. The real work starts after you submit it to Search Console: keep an eye on the gap between “submitted” and “indexed” counts. If the gap is large, it’s time to check the reasons in the indexing report.
The Indexing Layer: Is the Page Registered, and Under the Right Address?
Canonical: the single official address for the same content
If the same content can be opened from multiple URLs (http/https, with or without www, parameter variations), Google may treat them as separate pages and split the signals. The canonical tag declares the official address among the variations. The rule is simple: every page should point to its own clean URL as canonical, and all variations should point to that single address.
URL Inspection: an X-ray of a single page
Don’t guess at a page’s index journey — paste it into URL Inspection and look. The tool shows whether the page is indexed, when it was last crawled, which canonical was selected, and the reason for any block. For an important newly published page, the “request indexing” button can move discovery up. For definitions of indexing status terms, see the Search Console section of the glossary.
The Usability Layer: Speed, Mobile, and Security
Core Web Vitals are the three metrics Google uses to measure page experience, and they’re reported site-wide in Search Console:
| Metric | What it measures | ”Good” threshold |
|---|---|---|
| LCP | How fast the main content renders on screen | ≤ 2.5s |
| INP | Response time to interaction | ≤ 200ms |
| CLS | How much content shifts while loading | ≤ 0.1 |
These metrics don’t determine ranking on their own; they act as a tiebreaker among otherwise similar-quality results. In other words, speed optimization won’t rescue weak content, but it makes sure good content gets its due. In practice, the biggest LCP gain often comes from a single fix: compressing the page’s largest image and serving it in a modern format.
Mobile-friendliness has stopped being a separate checklist item — Google already indexes based on the mobile version. Sites offering rich content on desktop and a stripped-down version on mobile end up weakening their own index entry as a result. HTTPS isn’t even up for debate: a site without a certificate takes both a browser warning and a trust-signal hit.
Structured Data: Describing Content to Machines
Structured data is code that tags a page’s content using the schema.org vocabulary: this is an article, published on this date, by this author. Correct markup opens the door to star ratings, FAQs, and similar rich results in search — and a rich result means more clicks in the same position.
Use it within its limits: structured data can’t make content appear different from what it actually is. Marking up information that doesn’t appear on the page won’t earn you a rich result — it risks a manual action instead.
Summary
Technical SEO comes down to three questions: can Googlebot reach the page (robots.txt, sitemap, internal linking), is the page indexed under the correct address (canonical, noindex, URL Inspection), and is the page usable (Core Web Vitals, mobile, HTTPS)? If all three hold up, the technical side is doing its job; the rest of visibility depends on content quality and the authority built through external links.