Getting Indexed By Search Engines

After you build a blog or web site, you might notice that all that traffic you expected isn’t showing up and you’re not getting indexed by search engines. Perhaps you don’t see the hits coming in that you thought would be there.

Maybe you do see the traffic, but those AdSense ads are really looking like a waste of time now. No one seems to be clicking on them.

You, my friend, need to be indexed with the search engines better than you are, and it can sometimes take a long time for them to find your site and index it without a little encouragement from you.

If you are running AdSense on your site, you might not know that the traffic that comes from search engines is more likely to click an AdSense ad and will be more steady than social traffic will be.

People are always searching for something, and if your content answers what they are searching for, then at a minimum, you need to let the search engines know your alive and kicking. Otherwise, you might as well be invisible.

I’m going to show you how to stick your site right under the nose of the three major search engines so they will begin to index you, and get that ball rolling.

Once done, you just need to allow time to pass and let the search engines do their thing. That’s the beauty of this. For the most part, once you set your site up with Google, Yahoo! and MSN (Bing), you really won’t need to go back unless you need to make changes or just want to look at all the data your site is generating.

Make sure your site is ready

It’s important to note that if your site is still under development and not ready for full time traffic, you should wait until it is ready.

What does that mean? It means that your site is beyond the structural changes that you might want to make. It means that you are currently pumping out new content and have your categories established the way you want them (if running blog software) and any changes to be made are new additions and not modifications.

Things you want to make sure are completed before continuing:

1. Site is no longer under development. This means that you have completed your cross-browser [css] hacks, and all the fine tuning that should be done to your site is complete.
2. Robots.txt file is in place and accurately set up to reflect your sites unique structure. What is a robots.txt file? Google’s Webmaster Tools, described below, also contains a robots.txt checker tool that you can use to validate your robots.txt file.
3. Review your sitemap to ensure all pages you want indexed are accounted for, and any pages you don’t want indexed are not present
4. Ensure unique meta content is provided for each page, and all [seo] work is done.
5. Make sure page loading times are fast and within reason, and the content is edited and ready to be indexed.
6. For those without a custom domain name and with a brand new site, this is your last chance to make the change without negatively impacting your site. Once indexing begins, changing the root [url] used to access your site will mean all prior indexing will be for naught without redirects put in place, and that can be a time consuming effort. If you need to purchase a domain name, please feel free to buy one from me. It helps to pay my bills and keep my family afloat. Shameless, I know, but I think you probably understand, so thank you. Here are some ideas on finding a good domain name also.
7. Decide if you are going to use a www before your domain name or not. A site should be accessible by both www.somedomain.com and somedomain.com, however, search engines tend to view those as two different links, thereby causing a potential duplication of content issue. Make sure you decide to enforce either leaving off the www part, or enforce the inclusion of it.
8. Blogs should have at least 10 posts before you try to submit your site anywhere. Any less is just too easy to create, and search engines are smart enough to know better to include a blog that has obviously just started.

While it may not be harmful in any way to submit your site while there are still a few tweaks left to do, my advice is to have your site complete and ready for prime-time before contacting the search engines to let them know your site is alive and kicking.

Get indexed by search engines quickly

This article gives you the links you need to put yourself under the nose of these search engines. I’ve purposefully left out any links to directly “submit a link” to the engine since that is not always the best approach, and I personally avoid doing that. I’ve heard that you actually get penalized for doing so. So I simply avoid it altogether instead of tempting the fickle finger of fate.

However, the first thing you should do before anything else is get at least one link on a site that is already indexed by search engines. One favorite way that site owners have discovered is to place a link on Mixx or Digg, or even one of the many other social bookmarking sites that are out there.

The idea is that major search engines are already indexing Digg, Mixx, and others, so when they find a link to your site on one of these sites, and the search engine deems it as brand spanking new, the search engine will come visit and begin to index your content.

Call it a backdoor way to get indexed, doesn’t matter, it works and it’s perfectly legal in the eyes of search engines. I won’t steer your wrong.

Here is what you need to do to help get your site listed in the Google, Yahoo and MSN/Live/Bing search engines.

Preparing to submit your site to search engines

Each of the sites I’m going to discuss below will require that you have access to your source code. If you are in blogger, you’ll need to be able to access your template. If you are in WordPress, you’ll need to access the theme files.

You’ll be asked to verify that you own the site that you are setting up, and that is done by either:

1. Adding a file to the root of your site or
2. Adding a tag inside your homepage tag.

You’ll also want to supply a sitemap to each service if you happen to have one. All blogs will have a sitemap, and if you are not running blog software, you might not have one. If that is the case, you can use this tool to create a sitemap of your site. Note that if anything changes on your site, you’ll want to update the sitemap to reflect those changes.

With a sitemap in place, and connectivity to access and edit your source files, you’re ready to begin.

Setup your site in Google

Getting listed in Google is probably the easiest, and the best performer that I’ve seen in terms of traffic.

Google has three major applications that you should be using. Analytics, Webmaster Tools and Alerts.

It’s unproven, and I’m unable to find any discussion on the subject, but I believe that when you perform all three of these services, Google will deem you as serious about your site, and your site will benefit from having all three of these Google applications configured for your site.

Google Analytics – Traffic analysis data. Perhaps the most important aspect of operating a web site is knowing where your traffic comes from, and what type of visitors are accessing your site. Analytics will help you understand your visitors better. Visit this page for detailed info on establishing your site with Google Analytics.

Google Webmaster Tools – Ensures that Google can find all your content, it where you can supply Google with a sitemap, and will tell you what keywords are found within your content.

Google Alerts – Google Alerts are great for letting you know when Google indexes a particular keyword. In this case, I’m talking about your domain name keyword. For example, I would add ‘waynejohn.com’ to know when something linking to my blog has been indexed. It’s theorized by some that this also speeds up initial indexing of your site, and may be all you really need to do to get Google indexing you. Here’s an article that shows how I use Alerts.

Get your site into Yahoo!

I’ve noticed that if you run AdSense, Yahoo! tends to not index you as well, and your search traffic will suffer. Far be it for me to start a conspiracy theory about why, but you’ll probably notice this as I already have for a few of my sites, WayneJohn.com included.

There are 8 ways to submit your site information to Yahoo!, however, I’m of the opinion that one is good enough to get the ball rolling. However, this shouldn’t stop you from exploring the other options that Yahoo! provides.

Yahoo! has a Submit Your Site page that details numerous options for including your site in their indexes. Choose the ones that are appropriate to your site. At a minimum, you should use the Submit your site for free option (requires login). Here’s also a great article that walks you through this process.

Adding your site to MSN/Live/Bing

Even though many will say that you don’t need to submit a site to MSN/Bing since the majority of search traffic will come from Google first, and Yahoo! second, it’s still another search engine that can drive visitors to your site. So why limit yourself when there isn’t any harm in doing so?

To get indexed with MSN/Live/Bing, simply visit the Webmaster Center, add your site and sitemap.

Submitting to Ask

I’m going to throw in one more search engine as a bonus, and also because this one is incredibly easy to submit to.

To submit to Ask, simply copy this and modify it to include your domain name and path to your sitemap file.

http://submissions.ask.com/ping?sitemap=http%3A//www./

Replace with your actual domain name
Replace with the path to your site map

Once you modify the [url] to fit your site, simply pop it into a browser address bar and hit enter. Ask will then discover your sitemap, and will soon thereafter begin to index your content.

Conclusion

Getting indexed by search engines is the first step you need to take to start finding traffic for your site. Search engines will work day and night, 24/7 to help drive traffic to your site, so it makes sense to get your site into their cross-hairs soon after launch.

You’ll find that over time, and with better SEO efforts and keyword usage that the traffic generated from search engines will provide a constant flow of visitors to your site, unlike much social traffic that is here today and gone tomorrow.

If you found this article useful, you might want to subscribe to my feed. Also, why not help me out by sharing this with your friends by using the Tell a friend widget below or saving to your favorite social bookmarking site. Finally, please let me know your thoughts by leaving a comment below.

You should find getting your site indexed by search engines is an easy thing to do. Have fun waiting to see what happens with your site in these search engines now that they know you exist!

Update 6/2/2009 – Totally cheesed it and missed the fact that MSN is now Bing. I Sprinkled some Bing back into the post.

You might also enjoy these related posts

1. Where Does Web Site Traffic Come From? Where Does Web Site Traffic Come From? Web site traffic can come from a number of places, but these days there are five primary sources of traffic that you can...
2. The ebb and flow of Google search traffic One thing I’ve come to learn about search traffic is that it’s here today, gone tomorrow, and then it comes back again for no apparent reason. The ebb and flow...
3. 10 Ways To Increase Your AdSense Revenue So you have a blog or site, and you thought that by adding some AdSense ads to your site you would start to see some revenue from those ads, but...
4. Top 10 Search Engine Traffic Driving Content In light of my extended hiatus from posting here, here is some of my top 10 search engine traffic driving content found on WayneJohn.com.  While I’ve been away, I haven’t...
5. An overview Chitika search advertising for publishers In my quest for financial freedom, I’m very interested to learn more about how I might utilize the hundreds of posts I’ve written over the past two years.  AdSense has...