SEO – Link Building Series: How Search Engines find your blog or website?

How can I get my site on Google search?
I have answered that question at least 10 times on Yahoo Answers and several times over email. Instead of responding to yet another question individually, I thought of creating this reference post – and time permitting a series itself – here on this blog.

How people discover your blog or website?

my website on google searchBefore looking into the technical details on what it takes to increase the search visibility of your blog, one should understand how people discover your website content or blog posts on the web. Unless somebody already knew about your website, there are only three major ways for a total stranger to find you on the web and they are:

#1 Advertising: Online/offline advertising, Exchange advertisement programs, PPC (Pay-Per-Click) advertising, SEM (Search Engine Marketing), Article marketing etc

#2 Referral links: Blogrolls, Directory listing, Forums, RSS/Feed directories, Social bookmarks, Social network links etc

#3 Search Engines: Via Google, Yahoo, MSN, Ask etc.

Off the above three, the most popular method used by people to find information on the web is via search engines. It, probably, is also the most sustainable model for long term to drive traffic to you website or blog.

Now the process of optimizing your site content to make it search engine friendly and easier to spot by the search engine bots is what is known as Search Engine Optimization (SEO).

How Search Engines find information?

The World Wide Web (WWW) in principle is a huge repository of millions of interconnected PCs and information stored in them. So, in order to get your blog or website content to be probed and indexed by the search engines, they need to know about your siteโ€™s existence first. This can be done via search engine submission for newer sites or let the search engines automatically find your site via reference links to your site from other sites. In either case, ultimately the search engine spider (a piece of software that examines your site for its content) needs to crawl your site and index its content for the use of search users. Now, there are two entities on your site that helps these spiders probe your site effectively. They are:

#1 Robot.txt: Once your website is submitted to the search sites, the search engine spiders (also known as crawlers or bots) will attempt to navigate your site to understand its content. A robot.txt file maintained on your website will tell the spiders which all folders and modules on your web server are allowed to be crawled by them. (The format and composition of robot.txt is not in the scope of this post)

#2 Sitemap.xml: Once the robot.txt allows spiders to crawl certain areas of your server folders, the sitemap pitches in to provide more details. The sitemap.xml file contains all your pages, posts, links to tools and web applications etc that need to be made visible to the public users (and hence to be indexed by the search engines). Sitemap.xml will also contain more details on these pages such as when it was last changed, the priority of the page etc that help spiders to decide on when to index them.

Basically robot.txt provides folder/file level access details and sitemap.xml provides detailed information on pages. These two files are essential for any website that is search engine friendly and they are usually maintained at the root folder of your website.

Page & link level instructions to the spiders

In addition to the above to files, there are page and link level instructions that help spiders to decide whether to index a page or a target page that is pointed to by a link on the page (URL). These are page meta tags and link properties and I shall talk about them in another post soon.

Additional tips

  • Sitemap.xml is usually submitted to search engines via tools such as Google Webmaster Tools. Even robot.txt can be analyzed sample robot.txt created there in
  • Submiting your website or blog to search engines can be either done manually per search engine or done via bulk submission services (paid as well as free) offered by certain sites
  • Submission of your website does not necessarily guarantee that your site will be appearing in search results immediately. There are other aspects to prioritizing your site contentsโ€™ weightage

Part II: Page Rank and Search Engine Optimization >>

Comments

  1. Nihar :

    Ajith,

    Good post. This will definitely help guys who are stepping into blogging world.

    I did the steps you mentioned the very first day of publishing my first page. From next day, my website was appearing in Google Search Engine :)

    Nihar’s last blog post… January 2009 Blog Traffic & Income statistics

  2. qarla :

    First! to all newbies who managed to read this article, follow this one! Dollar shower is the best! :)

  3. osome post, really very useful to Growing bloggers like me, Thanks for sharing this cool post

  4. I wont be long and I’ll say directly what I did to make search engine discover my site time ago. When I first started blogging, this is what I did:

    Create the blog or site of course.

    Add original content

    Submit to Digg or similar sites

    Search engine scan them and come to your’s. On your side have a sitemap ready! Thats what I often reply when a friend ask me what to do to get his site listed quickly on google.

    Hope this little simple tips help someone :-) As for submitting the site to search engine… I dont really know…but I have the feeling that it just dont work. Had a very very very very old site lol. I submitted long ago and it took months before getting listed.

    The site is closed now though. The best is to get linked from high PR site as i said above then when google come to scan your’s, have your sitemap ready ๐Ÿ˜€

    Btw Ajith… I tried something just for fun this morning… and hell I dunno how I got nearly 1800+ visits today ๐Ÿ˜€ I normally dnt get above the 500 so I am feeling this is great lol. Cyclone warning in Mauritius here :S I hope the internet antennas dnt get blown away lol else I will have to live offline for a week :-(

    Kurt Avish’s last blog post… Gael Cyclone In Mauritius – The Details

    • What does ‘search engine scan them’ mean?

      • @Gillean, what Kurt probably meant is that instead of you submitting your blog directly to Google or other search engines, it’s good enough (or even better) if your posts get digged or stumbled. Since Google follows social media sites and indexes the content faster, that’s a shorter route to make the google bot visit your site. And once it visits, if it sees a good sitemap there, your entire content is indexed as well.

  5. Ajith,

    Great post. Will be really helpful to many beginners. But you really believe in “submitting of website via bulk submitting paid services”? That is strange…either I am missing something or typed it wrong.

    binaryday’s last blog post… 35 super useful wordpress plugins that you may ever need

  6. As a complete newbie to SEO, I appreciate you laying out the very first steps! Look forward to reading the next stages in helping my google rankings grow :)

    Lee.

  7. Donace :

    I think as I’ve mentioned before submitting sitemaps actually do provide a quick index time and boost in rankings (small yet there).

  8. Another great way to build links is through social media. I recommend your readers check out our comprehensive social media marketing plan to get started. Through social media, people build natural, qualified and relevant links when the write about what their passionate about.

    You can read it at:
    4-Step Action Plan for Social Media Marketing

    Thanks!

    ~Joe

  9. Hi, very nice article and also the comments by Kurt Avish are good too, i think these two articles can yeild far better results

  10. Very informative article, which I found quite useful. Cheers ,Jay

  11. Leo :

    This is a good pre-cursor to doing actual on-page optimization. I love the way you have explained how visitors can find you…wished I would have written this one first….:)

  12. @Nihar, thanks… Yes, the aim is to keep this series as reference posts so that I can make the beginners point to these pages whenever they come up with same queries

    @qarla, thanks a lot for your sweet words :)

    @Nicholas, thanks for your visit and comment. Please check back for the rest of the series.

    @Kurt, that is exactly what I meant when I said let the search engines automatically find your site via reference links to your site from other sites. These days popular bookmarking sites (and even ping services) drive search engine spiders our way. What I stressed on is the traditional way of doing it :) Thanks a lot for your description which is a very practical approach.

    @Binary, sorry for the confusing phrasing :) What I meant is that paid submission services that takes care of submitting your site to all search engines.

    @Lee, thanks buddy. Definitely, I will try my best to come up with practical tips during this series… I will be charged up by genuine demand :)

    @Donace, definitely… and also, the additional information on sitemap (how often content is changed, last changed date/time etc) is important as well. Otherwise, the spiders may not crawl for your subsequent changes.

    @Joseph, will talk about social media during this series when I touch upon link building.

    @khmohsin, thanks… yep, Kurt’s tip is really practical :)

    @Leo, thanks for your compliments. Since this series is for beginners, I will try to explain the content here in the simplest possible manner.

    Thanks folks for your comments ๐Ÿ˜‰

  13. Richael Neet :

    My first comment after a rather long time :)

    I usually ping my site updates to Google Blog search, Technorati and other such relevant websites. That helps me in getting my blog indexed faster. Moreover, social bookmarking (most effective being Digg) sometimes accelerates the site indexing frequency to mere hours…of course, speaking from experience.

  14. Dennis Edell :

    How on earth did you skip blog commenting?! ๐Ÿ˜‰

  15. gwen :

    This is very good information that you are providing here. I will be checking back to see what you will be including about link building. So I have you bookmarked. Thanks!

  16. @Richael, thanks for sharing your experiences… I will try to cover as much link building, seo tips as possible in the coming posts :)

    @Dennis, we haven’t started talking about link building yet :) just setting the pace…

    @gwen, thanks for your visit and comment. Please come back again to read the entire series

  17. Harish :

    Great post. I liked reading it. I have not done much of SEO as I am not much aware of it. May be this article will help me out a lot.

  18. robots and sitemap is very important for our sites … to get well into search engines …

    i had a probs with creating sitemap of my new site which is more then 15k … can u suggest any tool for it ?

  19. @Harish, will try to help beginners from the little I know :)

    Truck trader, have you tried Google sitemap generator script? If you have a blog, try Google XML Sitemaps plugin

  20. This is a great tutorial for beginners Ajith – now when people ask you questions you can just link to you own article (and get more traffic).

    As I was reading it I started thinking about terms that we use that are normal to us but are actually a little strange – robots, spiders – it almost sounds a bad horror film ๐Ÿ˜‰

    Ok – now I’m on to part 2.

    Kim Woodbridge’s last blog post… Adding WordPress 2.7 Threaded Comments to your Theme

Trackbacks

  1. […] SEO – Link Building Series: How Search Engines find your blog or website? -.. […]

Speak Your Mind

*