Lee Odden

SEO Tips – Let the Spider Crawl

Lee Odden     Online Marketing, SEO, SEO Tips

Over time I plan on adding more scheduled topics and features to Online Marketing Blog. The most recent has been the “Spotlight on Search” series of interviews. The newest is, “TopRank Basic SEO Tips” which will cover fundamental how-to’s on search engine optimization. The SEO tips topics will be motivated by client and prospect questions that our team answers daily.

So on to the first TopRank Basic SEO Tip: Let the Spider Crawl

For most sites, one of the first things we check is to make sure the site crawler friendly. “Crawler friendly” you say? What the heck does that mean? Search engines find sites mostly by following links from sites that are already known to find new sites and pages. The sofware programs that search engines use to perform this task are often called “Bots” or “Spiders”. You get the analogy right? “Web”, “Spiders”, “Crawl”.

If you don’t make sure your site is crawable and indexed, then you’re putting your web site at a gross disadvantage. For example, if you have a 1,500 page web site and only 700 pages are getting indexed, that’s like showing up to a baseball game with only 5 of your players. You need the whole team to win, so make sure your site is crawlable and getting indexed properly.

As search engine spiders crawl the links of your site, they make copies of the pages and then peform other functions that strip away the code, interpret the remaining text as well as other analysis that ultimately leads to a score for the page and association of the page to certain words. All of this along with links into your site from other web sites, influence your rankings. On the PPT slides from the recent Google Press Day, it says there are over 200 “signals” used to rank web pages on Google.

Here’s an animation of how Yahoo’s SLURP crawls a network of pages.

If a search engine has difficulty “crawling” the links within your site, then the pages either won’t get indexed at all or will only get partially indexed – neither of which will help your site’s rankings.

OK, now I know why, but what about the how? Search engine friendly URLs are simple. As in, short and simple. For example, the url of this web page is: https://www.toprankblog.com/2006/05/seo-tips-let-the-spider-crawl/

It could be something like https://www.toprankblog.com/?pageid=234234&articleid=5tips&postid=435345 or something similar. The second url is still crawlable, but if you got to pick, which one would you prefer to index? Which one would you be more likely to remember as a user?

Most problems with links and the URLs they point to getting crawled involve shopping cart software or content management systems that place a lot of extra information in the web page URL. If references to “?sid=” or a large number of variables are included in the URL it can cause issues. Search engine bots are leary of “spider traps” or situations with calendars or where an infinite number of url versions display the exact same web page. This often occurs with the use of session ids.
Simple and short urls are typically the easiest to crawl so try to use a content management system that produces short, clean URLs.

You can also use programs like Google Sitemaps to submit your site URLs for inclusion. There is no guarantee it will work, but it’s been pretty effective for many web sites. Google Sitemaps works in conjunction with a normal “crawl” of your web site. Plus there are many useful troubleshooting features and information available with Google Sitemaps. You can also submit an RSS feed or plain text file of your site’s URLs to Yahoo.

There’s actually quite a bit more involved with making your site crawlable, but I’ll leave it at this for now.

Resources on crawler friendly web sites:

PoorSo SoOKGoodAwesome (1 votes, average: 4.00 out of 5)

Lee Odden About Lee Odden

@LeeOdden is the CEO of TopRank Marketing and editor of Online Marketing Blog. Cited for his expertise by The Economist, Forbes and the Wall Street Journal, he's the author of the book Optimize and presents internationally on B2B marketing topics including content, search, social media and influencer marketing. When not at conferences, consulting, or working with his talented team, he's likely running, traveling or cooking up something new.


  1. Lee … very nice intro to spidering. I am all for educating people about SEO so that people do not fall into a SEO Myth or Hype trap.

    I was actually told by one SEO company that using Yahoo paid inclusion, you can pay to be at the top 10 natural listings. Same thing would work for Google (in their words).

    How unfortunate that this is a company you know definitely and we all visit their booth in the shows.

  2. Hey Igor, yeah it’s not a good thing when sales people for SEO firms get carried away.

  3. Hi Lee, good call with the variables in the url. Some other spider traps I think of are javascript and flash. Especially when they’re used in the nav bar. Other possible spider traps include robots.txt., internal links pointing at dynamic pages, duplicate content issues, AJAX, and frames. Possible solutions are url rewrites, and buying links to deep pages within your site, and Google Sitemaps, and the obvious rewriting content especially if it’s in the title’s & descriptions.

  4. Where should we put our logos and text links to inner pages in search engine point of view.
    how good for a site to put text links instead of image links?


  1. […] some vintage crawler SEO advice, check out this post on improving site spidering from 2006.             Subscribe to […]