Search Engines Build

Search Engines Build

How search engines build their index

Most well-known search engines like Google and Bing have trillions of pages in their search indexes. So before we talk about ranking algorithms, let’s drill deeper into the mechanisms used to build and maintain a web index.

Let’s break this down, step by step:

• URLs
• Crawling
• Processing & rendering
• Indexing

Sidenote. The process below applies specifically to Google, but it’s likely very similar for other web search engines like Bing. There are other types of search engines like Amazon, YouTube, and Wikipedia that only show results from their website.

Step 1. URLs

Everything begins with a known list of URLs. Google discovers these through various processes, but the three most common ones are:

From Backlinks

Google already has an index containing trillions of web pages. If someone adds a link to one of your pages from one of those web pages, they can find it from there.

From Sitemaps

Sitemaps list all of the important pages on your website. If you submit your sitemap to Google, it may help them discover your website faster.

From URL Submissions

Google also allows submissions of individual URLs via Google Search Console.

Step 2. Crawling

Crawling is where a computer bot called a spider (e.g., Googlebot) visits and downloads the discovered pages.

It’s important to note that Google doesn’t always crawl pages in the order they discover them.

Step 3. Processing

Processing is where Google works to understand and extract key information from crawled pages. Nobody outside of Google knows every detail about this process, but the important parts for our understanding are extracting links and storing content for indexing.

Google has to render pages to fully process them, which is where Google runs the page’s code to understand how it looks for users.

That said, some processing occurs before and after rendering—as you can see in the diagram.

Step 4. Indexing

Indexing is where processed information from crawled pages is added to a big database called the search index. This is essentially a digital library of trillions of webpages where Google’s search results come from.

That’s an important point. When you type a query into a search engine, you’re not directly searching the internet for matching results. You’re searching a search engine’s index of web pages. If a web page isn’t in the search index, search engine users won’t find it. That’s why getting your website indexed in major search engines like Google and Bing is so important.