Lee Odden

SEO Basics: Unlocking Content for Search Engines

Lee Odden     Online Marketing, SEO, SEO Tips

At the DMA08 conference in Las Vegas this week, the “Search Engine Experience” panel on Tuesday was presented with a number of site reviews which are great sessions because the audience gets free SEO consulting advice and speakers get to demonstrate their expertise.

One of the questions that came up was related to a site that has most of it’s content behind a login. Because search engine spiders don’t/can’t fill out forms (with some exceptions) they cannot find the content behind the login to index and include in the search results.

Why would you want your “members access only” content available to search engines?

The more web pages a search engine can find, index and include in the search results, the more entry points there are for new customers to find your web site content. A broader, yet relevant, footprint on the web can make a significant difference in traffic to a web site. If most of the content is hidden behind a login process, that content is also hidden from the search engines.

Of course there are a number of good reasons for web sites to put their content behind a login. The site in question at DMA08 concerned an auction site that involved being a member to access the listings. Other examples include content that is sensitive, industry regulated or that is time sensitive.

In the case of the auction site, two issues were involved: most of the site was behind a login and the auction content was only relevant for a certain period of time.

A common solution for making content hidden behind a login available to search engines is to modify the templates that pull data from the database so that an excerpt is pulled into a crawlable web page. The excerpt content and links between them would be available to search engine spiders and get included in search results. Search visitors that find the excerpts would be presented with a link to read more of the article/content and then be presented with an option to become a member, register or whatever the criteria for access are.

In the case of the auction site, outdated content would be of no use to the customers so it makes little sense to allow it to be indexed by search engines. Sitting in the audience, I was doing my best to mind meld with the panel (Heather Lloyd-Martin, Detelv Johnson and Jeannette Kocsis) a suggestion but Detlev already had it:  The auction site could create content summaries such as the top bids for a category or pictures and descriptions of the most popular items. A blog talking about the products being sold might also be of use.

When content is regulated by an industry, it’s obviously important to work with the compliance “powers that be” within an organization to determine how much information can be made publicly available. Membership sites also need to gauge how much information to make public before asking readers to sign up for more.

A site with thousands of web pages can see a dramatic increase in pages indexed when using an excerpt strategy. However, it will take some convincing of IT/web development to make the necessary changes in the content management system and database, if that’s even possible. Then there’s the cost justification.

To build a business case for making these kinds of programming changes requires some forecasting of potential traffic increases which shouldn’t be too difficult using existing historical web analytics. Some SEO consultants are fairly good at this and can help you out. 🙂

If your web site has undergone the programming changes in order to make some of your previously hidden content available to search engines, it would be great to hear about it. What were the biggest hurdles in getting budget, implementation and even how it panned out once the content was live.?

PoorSo SoOKGoodAwesome (2 votes, average: 2.00 out of 5)

Lee Odden About Lee Odden

@LeeOdden is the CEO of TopRank Marketing and editor of Online Marketing Blog. Cited for his expertise by The Economist, Forbes and the Wall Street Journal, he's the author of the book Optimize and presents internationally on B2B marketing topics including content, search, social media and influencer marketing. When not at conferences, consulting, or working with his talented team, he's likely running, traveling or cooking up something new.


  1. This artcle is very good. I am running a forum and blog site but when I allow people to read my technical articles and blogs they don’t like to join and participate in the forum. So I usually like to have the restriction in place. Can you sugegst some other thing? My site is http://www.techcubetalk.com (TechCuBe Talk – The Tech People)