Why Isn’t My Website Showing Up on Google?
This is a question we frequently get asked after launching a new website. To answer this question, let’s dig into how Google search indexing works and how websites interact with search engines.
The Search Index
A search index is basically a catalogue of all the websites that are on the internet. It is a giant database containing information about a website such as a website’s address (also known as the domain or URL), number of pages, page titles, page descriptions, and plenty of other information.
When you type a search into a search engine (such as Google), the search engine references this giant database and returns a list of results. So, for your website to be included in the list of search results, your website needs to be “indexed” or added to the search engine’s database.
Web Crawlers
All major search engines have a program (called a bot) that “crawls” the internet and looks for new websites, or changes to websites and uses that information to add to or update its search index. Googlebot is the name of Google’s crawler, and it runs around automatically scanning websites. As a website owner or a website developer, you have no control over these bots’ programming and how often they interact with your site. The only thing you have control over is the messages you put on your website for these bots.
The Robots.txt File
We can send messages to these bots such as “Please don’t crawl my site!” or “Yes! I want to be in your index!” To do this, we use a text file, named robots.txt. At Iceberg, while we are developing a website, we change this file to say, “Please don’t crawl my site,” because we don’t want any of the in-progress content to get indexed.
Once the website has been launched, we change this file to say, “Yes! I want to be indexed!”
WordPress allows for easy changes to this file by using a checkbox that says, “Discourage search engines from indexing this site.” If the box is checked, the robots.txt file is edited to disallow search engines from scanning the site. It is up to the search engines to honor this request (most do).
When the box is unchecked, the file says allow, and the site is open for indexing. The next time a bot scans the site it will index it.
But I Don’t Want to Wait!
You may be wondering how to speed up the process of getting your site found by a bot. Unfortunately, we are at the mercy of the crawling program, in addition to the number of machines throughout the internet that are running the bot simultaneously. With some search engines, you can tell them, “Hey! Look over here! I have a new site I want to be indexed!”
Google Search Console is a tool that lets you measure and monitor your website’s presence in Google’s search index. When you connect your website to Google Search Console, you can use it to submit a sitemap and request indexing. At Iceberg, when we launch a new website, part of our process includes editing the robots.txt file, connecting the site to Search Console, and requesting indexing. If you are anxious for your new website to show up at the top of search results immediately, most search engines will allow you to purchase advertising.
If you want to maximize your presence in search engines, that’s where Search Engine Optimization (SEO) comes in. SEO can be done directly on your website, through technical enhancements, speed and image optimization, and keyword-rich content generation. SEO can also be performed off-site, by utilizing third party websites to drive traffic to your business, such as social media and online directories. Iceberg offers a number of SEO services aimed at increasing your website’s visibility.