Crawling & indexing — How Google finds your pages
How search engines discover your pages (crawling) and store them to show in results (indexing).
What it means
Before Google can show your page in results, it must find and understand it. This happens in two steps.
Crawling is when search engine programs, often called bots or spiders, visit web pages and follow links to discover new and updated content. Google's bot is called Googlebot.
Indexing is when Google analyses a crawled page and stores it in its huge database, called the index. Only indexed pages can appear in search results.
How it works
Googlebot finds pages mainly by following links, both from other websites and from within your own site. You can also help by submitting an XML sitemap, which is a file listing the important pages on your website, through Google Search Console.
After crawling, Google decides whether to index the page. It may skip pages that are low quality, duplicates of other pages, or blocked by settings such as a "noindex" tag.
In Search Console, the URL Inspection tool lets you check whether a specific page is indexed, see any problems, and request indexing after you publish or update it.
A simple example
A Kota coaching institute launches a NEET crash course page. A week later, it still does not show up on Google.
The marketer opens Search Console and uses URL Inspection. It shows the page is "not indexed" because no other page links to it. She adds a link from the homepage and the courses menu, updates the XML sitemap, and requests indexing. Soon the page starts appearing in results.
Why it matters
If your pages are not indexed, they cannot rank, however good they are. Checking indexing is often the first step when a page is missing from Google.
Beginner tips
- Set up Google Search Console for every website you manage.
- Submit an XML sitemap and keep it updated.
- Link to new pages from existing pages on your site.
- Common mistake: leaving a "noindex" setting switched on after a site goes live.