SEO – Link Building Series: How Search Engines find your blog or website?
Ajith | Feb 2009 | Site Promotion, SEO
How can I get my site on Google search?
I have answered that question at least 10 times on Yahoo Answers and several times over email. Instead of responding to yet another question individually, I thought of creating this reference post – and time permitting a series itself – here on this blog.
How people discover your blog or website?
Before looking into the technical details on what it takes to increase the search visibility of your blog, one should understand how people discover your website content or blog posts on the web. Unless somebody already knew about your website, there are only three major ways for a total stranger to find you on the web and they are:
#1 Advertising: Online/offline advertising, Exchange advertisement programs, PPC (Pay-Per-Click) advertising, SEM (Search Engine Marketing), Article marketing etc
#2 Referral links: Blogrolls, Directory listing, Forums, RSS/Feed directories, Social bookmarks, Social network links etc
#3 Search Engines: Via Google, Yahoo, MSN, Ask etc.
Off the above three, the most popular method used by people to find information on the web is via search engines. It, probably, is also the most sustainable model for long term to drive traffic to you website or blog.
Now the process of optimizing your site content to make it search engine friendly and easier to spot by the search engine bots is what is known as Search Engine Optimization (SEO).
How Search Engines find information?
The World Wide Web (WWW) in principle is a huge repository of millions of interconnected PCs and information stored in them. So, in order to get your blog or website content to be probed and indexed by the search engines, they need to know about your site’s existence first. This can be done via search engine submission for newer sites or let the search engines automatically find your site via reference links to your site from other sites. In either case, ultimately the search engine spider (a piece of software that examines your site for its content) needs to crawl your site and index its content for the use of search users. Now, there are two entities on your site that helps these spiders probe your site effectively. They are:
#1 Robot.txt: Once your website is submitted to the search sites, the search engine spiders (also known as crawlers or bots) will attempt to navigate your site to understand its content. A robot.txt file maintained on your website will tell the spiders which all folders and modules on your web server are allowed to be crawled by them. (The format and composition of robot.txt is not in the scope of this post)
#2 Sitemap.xml: Once the robot.txt allows spiders to crawl certain areas of your server folders, the sitemap pitches in to provide more details. The sitemap.xml file contains all your pages, posts, links to tools and web applications etc that need to be made visible to the public users (and hence to be indexed by the search engines). Sitemap.xml will also contain more details on these pages such as when it was last changed, the priority of the page etc that help spiders to decide on when to index them.
Basically robot.txt provides folder/file level access details and sitemap.xml provides detailed information on pages. These two files are essential for any website that is search engine friendly and they are usually maintained at the root folder of your website.
Page & link level instructions to the spiders
In addition to the above to files, there are page and link level instructions that help spiders to decide whether to index a page or a target page that is pointed to by a link on the page (URL). These are page meta tags and link properties and I shall talk about them in another post soon.
Additional tips
- Sitemap.xml is usually submitted to search engines via tools such as Google Webmaster Tools. Even robot.txt can be analyzed sample robot.txt created there in
- Submiting your website or blog to search engines can be either done manually per search engine or done via bulk submission services (paid as well as free) offered by certain sites
- Submission of your website does not necessarily guarantee that your site will be appearing in search results immediately. There are other aspects to prioritizing your site contents’ weightage
Ajith,
Good post. This will definitely help guys who are stepping into blogging world.
I did the steps you mentioned the very first day of publishing my first page. From next day, my website was appearing in Google Search Engine
Nihar’s last blog post… January 2009 Blog Traffic & Income statistics
First! to all newbies who managed to read this article, follow this one! Dollar shower is the best!
osome post, really very useful to Growing bloggers like me, Thanks for sharing this cool post
I wont be long and I’ll say directly what I did to make search engine discover my site time ago. When I first started blogging, this is what I did:
Create the blog or site of course.
Add original content
Submit to Digg or similar sites
Search engine scan them and come to your’s. On your side have a sitemap ready! Thats what I often reply when a friend ask me what to do to get his site listed quickly on google.
Hope this little simple tips help someone
As for submitting the site to search engine… I dont really know…but I have the feeling that it just dont work. Had a very very very very old site lol. I submitted long ago and it took months before getting listed.
The site is closed now though. The best is to get linked from high PR site as i said above then when google come to scan your’s, have your sitemap ready
Btw Ajith… I tried something just for fun this morning… and hell I dunno how I got nearly 1800+ visits today
I normally dnt get above the 500 so I am feeling this is great lol. Cyclone warning in Mauritius here :S I hope the internet antennas dnt get blown away lol else I will have to live offline for a week 
Kurt Avish’s last blog post… Gael Cyclone In Mauritius – The Details
What does ‘search engine scan them’ mean?
@Gillean, what Kurt probably meant is that instead of you submitting your blog directly to Google or other search engines, it’s good enough (or even better) if your posts get digged or stumbled. Since Google follows social media sites and indexes the content faster, that’s a shorter route to make the google bot visit your site. And once it visits, if it sees a good sitemap there, your entire content is indexed as well.
Ajith,
Great post. Will be really helpful to many beginners. But you really believe in “submitting of website via bulk submitting paid services”? That is strange…either I am missing something or typed it wrong.
binaryday’s last blog post… 35 super useful wordpress plugins that you may ever need
As a complete newbie to SEO, I appreciate you laying out the very first steps! Look forward to reading the next stages in helping my google rankings grow
Lee.
I think as I’ve mentioned before submitting sitemaps actually do provide a quick index time and boost in rankings (small yet there).
Another great way to build links is through social media. I recommend your readers check out our comprehensive social media marketing plan to get started. Through social media, people build natural, qualified and relevant links when the write about what their passionate about.
You can read it at:
4-Step Action Plan for Social Media Marketing
Thanks!
~Joe
Hi, very nice article and also the comments by Kurt Avish are good too, i think these two articles can yeild far better results
Very informative article, which I found quite useful. Cheers ,Jay
This is a good pre-cursor to doing actual on-page optimization. I love the way you have explained how visitors can find you…wished I would have written this one first….:)
Leo’s last blog post… Take Back Your Life…How to Get More Traffic to your Site With Just One Post a Week..
@Nihar, thanks… Yes, the aim is to keep this series as reference posts so that I can make the beginners point to these pages whenever they come up with same queries
@qarla, thanks a lot for your sweet words
@Nicholas, thanks for your visit and comment. Please check back for the rest of the series.
@Kurt, that is exactly what I meant when I said let the search engines automatically find your site via reference links to your site from other sites. These days popular bookmarking sites (and even ping services) drive search engine spiders our way. What I stressed on is the traditional way of doing it
Thanks a lot for your description which is a very practical approach.
@Binary, sorry for the confusing phrasing
What I meant is that paid submission services that takes care of submitting your site to all search engines.
@Lee, thanks buddy. Definitely, I will try my best to come up with practical tips during this series… I will be charged up by genuine demand
@Donace, definitely… and also, the additional information on sitemap (how often content is changed, last changed date/time etc) is important as well. Otherwise, the spiders may not crawl for your subsequent changes.
@Joseph, will talk about social media during this series when I touch upon link building.
@khmohsin, thanks… yep, Kurt’s tip is really practical
@Leo, thanks for your compliments. Since this series is for beginners, I will try to explain the content here in the simplest possible manner.
Thanks folks for your comments
My first comment after a rather long time
I usually ping my site updates to Google Blog search, Technorati and other such relevant websites. That helps me in getting my blog indexed faster. Moreover, social bookmarking (most effective being Digg) sometimes accelerates the site indexing frequency to mere hours…of course, speaking from experience.
Richael Neet’s last blog post… .Pro TLD – A brief Introduction
How on earth did you skip blog commenting?!
Dennis Edell’s last blog post… Free Backlinks: A Traffic Experiment via Commenting & Promotion
This is very good information that you are providing here. I will be checking back to see what you will be including about link building. So I have you bookmarked. Thanks!
@Richael, thanks for sharing your experiences… I will try to cover as much link building, seo tips as possible in the coming posts
@Dennis, we haven’t started talking about link building yet
just setting the pace…
@gwen, thanks for your visit and comment. Please come back again to read the entire series
Great post. I liked reading it. I have not done much of SEO as I am not much aware of it. May be this article will help me out a lot.
[...] SEO – Link Building Series: How Search Engines find your blog or website? -.. [...]
robots and sitemap is very important for our sites … to get well into search engines …
i had a probs with creating sitemap of my new site which is more then 15k … can u suggest any tool for it ?
@Harish, will try to help beginners from the little I know
Truck trader, have you tried Google sitemap generator script? If you have a blog, try Google XML Sitemaps plugin
This is a great tutorial for beginners Ajith – now when people ask you questions you can just link to you own article (and get more traffic).
As I was reading it I started thinking about terms that we use that are normal to us but are actually a little strange – robots, spiders – it almost sounds a bad horror film
Ok – now I’m on to part 2.
Kim Woodbridge’s last blog post… Adding WordPress 2.7 Threaded Comments to your Theme
@Kim, thanks and you spelt out my idea
I think I can get people point to this post from where they can learn a few basics on SEO and link building.
I think pinging is another way to let search engines find and index your page.
But sometimes search engines can find your site with some other secret methods which is not listed above, Recently I have built a new website and did not have started link building or any other SE marketing and I found it is indexed by google, strange?!
Chinese Girl’s last blog post… Ramada Nanjing
@Chinese Girl, thanks for your visit and comment
Thanks for reminding me of ‘pinging’ as a means to alert the search engines to crawl your site. Platforms like WordPress has this feature built-in and plugins like Google XML sitemap, further enforces it.
Even without doing search engine submission, the search engine bots could still find you from your public profiles (on social networks, yahoo/Google groups, blog directories, FeedBurner etc)
We generally don’t bother submitting our sites to the search engines as such. We do an initial batch of directory listings and links from quality high PR sites, allied to good on-page optimization, and we find the pages get indexed quickly by the search engine spiders. It actually seems to take longer ifthe site gets submitted….
One of my friends, Swastik has really found an ingenious method of quickly getting your site indexed in Google. Just go to Google Keyword Tool, analyze your website for keywords and presto; your site gets crawled and indexed in a very short time. I have tried this and it seems to work for me too.
Just a little tip I felt like sharing…
@Ruby web, that sounds like a strategy
@Richael, wow, cool tip… I should try it with my very next post
Submitting to web directories is a vital part of every successful link building strategy. Apart from driving traffic to your website through direct referrals, web directories provide static, one-way links to your site, boosting your link popularity and improving your rankings on the major search engines like Google and Yahoo.
Sometimes adding the sitemap.xml and robots.txt is just not enough. If it’s a fresh website I would not recommend adding the project to the directories, as this will not bring you a desired effect, Google just will see another link in the spam directory with low authority rate. Instead, I would suggest adding the website to the social media platforms, like twitter f.e. Since Google have launched the real-time search, he has to index this information, so all the links posted in twitter are crawling almost immediately, and from my experience are indexed faster, even if they have a tag.
sir pls tell correct use robot.txt.i don’t know robot.txt but my site restricted robot.txt so pls send infn how to recover this problem when this robot.txt file will useful one and send infn seo
Best way to solve the issue is to sign up with Google Webmaster Tools, upload your sitemap and diagnoze your robot.txt file there. It has a mechanism to design and validate robot.txt file. Otherwise, just contact me with your blog URL and current robot.txt content.
I just checked your blog. You seem to have disabled the search engines from finding your blog. First you check your blogger admin settings. In Settings -> Basics, opt for “Yes” to ‘Let search engines find your blog” and “Add your blog to our listings”. Once that is done, search engines should find your blog and index it.
In addition (after making the above change) load your blog in the browser and choose “View source” from browser and search for “<meta content=’noindex’ name=’robots’>”. If that statement is there, search engines will not index your posts. In that case, mostly you will have to edit the blogger template itself to remove it.
sir i doing this step now how to finding my blog in search index pls help me sir
@Anbu, Your site is already indexed by Google. You will surely start building traffic in the due course of time. You need to do a little bit of site optimization to create the right post titles though.
sir my site now also having robot.txt now if it’s affect my search result and how to modify the robot.txt file