What Google Is Actually Doing

What Google Is Actually Doing

When a business owner says "we don't rank," they almost always mean one thing: they published a page, they want it near the top, and it is not there. But ranking is the last step in a four-part pipeline, and most "we don't rank" problems are actually failures earlier in that pipeline. Google has to find your page, load it, decide it is worth keeping, and only then decide where it belongs for a given search. If any of the first three steps breaks, the fourth never happens. Understanding the whole machine is what separates fixing the problem from guessing at it.

Step 1: Discovery and crawling

Google cannot rank a page it has never seen. It finds URLs three main ways: by following links (internal links from your own pages and external links from other sites), by reading your XML sitemap (a machine-readable list of every page you want found), and by receiving IndexNow pings that tell search engines a URL just changed. A clean sitemap is the fastest, most reliable way to hand Google your full list of pages, which is exactly what the XML sitemap tool exists to keep tidy.

The gatekeeper for all of this is robots.txt, a small file at the root of your domain that tells crawlers where they may and may not go. A single wrong line (for example, Disallow: /) can block your entire site, and it happens more often than you would think, usually left over from a staging environment.

For most local businesses, crawling is effortless: a dentist with 40 pages gets crawled top to bottom with room to spare. Crawl budget only becomes a real constraint on large sites. If a multi-location HVAC company publishes 5,000 city-and-service pages, Google will not crawl all of them on every visit. It allocates a rough budget based on your site's size, health, and authority, then spends it on the URLs it thinks matter. Waste that budget on junk (endless filter URLs, broken redirects, slow responses) and your important pages get crawled less often.

Here is the part people get wrong. What silently blocks crawling is not a noindex tag. Noindex is an indexing instruction, and Google must crawl the page to read it. The things that actually stop the crawler are:

• robots.txt disallows that fence off directories or the whole site.
• nofollow on links, which tells Google not to pass through them.
• Orphan pages with zero internal links pointing at them, so nothing leads the crawler there.
• Server errors (5xx responses, timeouts) that make Google back off to avoid overloading you.

The site scanner surfaces exactly these blockers so you are not hunting blind.

Step 2: Rendering

Once Google fetches your HTML, it renders the page much like a browser would, running your JavaScript and building the final layout before it judges anything. This matters because rendering can be a two-step process. Google may index your raw HTML first, then come back later to render the JavaScript when resources free up. If your core content, links, or headings only appear after client-side JavaScript runs, you can wait days or weeks for that second pass, and until then Google effectively sees a half-empty page. Heavy client-side frameworks are the usual culprit. The safest posture is to make your primary content present in the HTML, not assembled in the browser.

Step 3: Indexing

Rendering done, Google decides whether the page is worth storing in its index and files it by topic. This is a quality gate, not a formality. Plenty of crawled pages never make it in.

A big part of indexing is canonicalization. When Google sees several near-identical pages (a product with tracking parameters, http and https versions, a printer-friendly copy), it picks one canonical URL to represent the group and sets the rest aside. If you do not tell it which one you prefer, it chooses for you, and it sometimes chooses wrong. Controlling those signals cleanly is the job of your titles, canonicals, and robots directives.

Pages get left out of the index for a handful of repeatable reasons: thin content with little unique value, duplicates folded into a canonical, an intentional noindex tag, or blocked resources that stop Google from rendering the page fully. In Google Search Console you will see two states worth memorizing:

• Discovered - currently not indexed. Google knows the URL exists but has not crawled it yet. On large sites this usually signals a crawl-budget or priority problem: Google is not convinced the page is worth spending a crawl on.
• Crawled - currently not indexed. Google crawled the page and chose not to keep it. This is almost always a quality signal: thin, duplicative, or too similar to something already indexed.

Those two labels tell you where in the pipeline you are stuck, which is why connecting the account early matters. Reading your real index coverage in Search Console turns "we don't rank" into a specific, fixable diagnosis.

Step 4: Ranking

Only now does ranking happen. For a given search, Google scores your indexed pages using hundreds of signals that roll up into three questions: is this page relevant to what was asked, is it high quality, and does the context fit this specific user? Natural-language models handle query understanding, interpreting messy real-world phrasing rather than matching keywords literally. Results are personalized by location and history, which is why a roofer ranks differently in one suburb than the next and why so much local visibility plays out in the map pack. Freshness gets weighted up for time-sensitive queries ("storm damage roof repair today") and matters little for evergreen ones.

The helpful-content signal is no longer a separate periodic update. It has been folded into Google's core ranking systems, which means "was this written to help a person or to game a search engine" is now judged continuously, page by page, as part of everyday ranking.

The throughline

A perfect title tag on a page Google cannot index is worth exactly nothing.

Fix crawl and index before you chase ranking tweaks. Rewriting headings, adding keywords, and polishing meta descriptions all assume the page is in the index and eligible to compete. Confirm that first. Get discovery, rendering, and indexing clean, keep them clean, and ranking work finally has something to act on. That order is the whole discipline of technical SEO.

Related lessons

• Connect Search Console. See your real index coverage and which pages are stuck at "discovered" or "crawled but not indexed."
• The site scanner. Surface the crawl and index blockers on your own site before they cost you rankings.
• Titles, canonicals, and robots. Control the exact indexing directives that decide whether a page gets kept.
• The sitemap system. Understand how your full page list is handed to Google for discovery.
• Technical foundation. The ongoing work that keeps crawl and index healthy so ranking effort is not wasted.