Although search engine indexing and crawl patterns don't always correlate to search engine rankings, webpages that rank well in competitive search engine query spaces always have solid indexing and crawl patterns. Optimizing link structures and architecture for both search engine indexing/crawl as well as users can be a challenging task.

I've always leaned toward optimizing for users and have found that more often than not; optimizing for robots typically follows suit. But that's not always the case. When working with large websites a search marketer is presented with a challenge that small site optimization typically doesn't have to struggle with as much; enhancing crawl rates from search bots and presenting numerous webpages that are both unique and indexable.

Enhancing crawl rate and crawl depth can be accomplished in a few different ways depending on the size and nature of the website. The first way is within the source of the website. Websites driven with a CMS typically tend to pull and parse databases inefficiently and create extraneous functions and controllers, and I would not recommend them to anyone hosting a website with over 10,000 pages.

Pure PHP and ASP can be a beautiful thing. I've been fortunate to witness with my own eyes the effects of intelligent, clean, and efficient code structures and how they can optimize GWT download rates and (consequently) indexing patterns for a website. Whether they are simply grabbing a page or performing a deep crawl, the less time a search bot spends waiting for sockets to open on your website, the better. This time can obviously be cut down by hosting improvements but it can also be cut down, literally by the millisecond, by having an efficient code structure in place.

There are a lot of different robots out there that do a lot of different tasks when opening the socket of your website, and a lot of what they do depends on how complex your website is. How intrigued they become depends on how unique and complex your webpages are and how efficiently you've mapped your website.

US Patent 7,383,254 from Microsoft tells us that a search engine can easily distinguish (and has been able to distinguish as early as 2005) parts of a webpage. That means boilerplates, navigational blocks, and repeated or unique content all get weighted and scored differently by a search bot. Although I'm not going to dive into block style analysis of webpages right now, I am going to use the example to point out that the nature and location of your internal links can have a dramatic effect on your crawl.

Classical SEO ideology tells us that the fewer 'hops' away your deep-rooted pages are from the root directory, the better optimized the site is for a search bot. Generally speaking, I would agree with that statement, but it does leave a lot of ground still to be covered.

The first thing I do for a client when asked to optimize link patterns and architecture is consider what pages on the site have the most potential for landing and what pages can contribute to increased conversion. A lot of times I get the same results for both questions. Once the landing hubs, or silos, or whatever you'd like to call them are realized, we can create link patterns that emphasize them.

An important question to ask is how many entry points does a search bot have on your webpages? Remember, entry points can be both internal and external - and a good external entry point can be worth more than 10 internal entry points. This is because in a typical crawl - a bot extends deeper into the website from the root URL. Extending to a page internally might take some time, but an external entry point might provide a much needed shortcut.

Within a classic 3-tier website, a search bot might find the root URL, grab 50 links and begin its crawl on layer B. Once they get to layer B they might grab 70 links from 50 pages and crawl those 3500 links to begin working on layer C. This would be a very time consuming process as we can assume the bots will crawl the layer linearly rather than simultaneously. This is not what I would consider to an optimal strategy for getting pages that change frequently on layers B and C crawled often.

A more optimal environment would have multiple points of entry for external sources (links) to layer B (which links to layer C) or layer C itself (which might interlink with layer C). You can see visually how this would cut your wait times down on crawling:

A good way to analyze the entry point efficiency within an existing website with history and deep pointing external backlinks is to use a script that pings internal and external links pointing to a page, and also calling how many pointing links the page itself has:

Assuming that Webpage A is our root URL, and ignoring tier structure altogether, let's call webpage F our "money page" - that is, we'd like to see that indexed, ranked, and drawing traffic. This is because webpage F can either:

• Convert traffic directly on the page, or
• Point landing traffic to other avenues of conversion quickly and easily.

Webpage F is also topically relevant to rank in query space. So what do we do? Emphasize Webpage F with internal link structure where it makes sense. The higher the number of pointing internal links to F, the better. We'd also try and build out external backlinks.

Not rocket surgery, right?

But what if Webpage F is not a good candidate for acquiring external backlinks? We can still build out internal navigation to emphasize this page. Creating unique and dynamic link structures (i.e. not boilerplate or block navigation) back to Webpage F from externally backed pages within our website would do almost an equally good job of this.

Why is this? Because a search bot might have a lot of external entry points on webpage B, and if those external entry points are crawled frequently (from RSS feeds, being close to the root, etc), even though webpage B might be 4 hops away from webpage A (our root URL), it would get crawled frequently because of this.

I consider search engine bots to devalue link blocks and boilerplate navigation, so to funnel emphasis from webpage B to webpage F, I would create interlinking within content, image links - basically anything that is unique from the way the rest of the pages are interlinked and makes sense from a usability perspective. Of course optimal anchor and visual location of the link would also play a role here.

Taking this a step further, there's also been evidence that search engines can analyze parts of a website. They can determine, within a categorical scope, what a piece of a website is about. I've seen this myself with new website launches and testing behaviors from Google:

They will begin indexing and ranking section A, then ranking section AB (a subsection of A) and then drop AB so they can index and rank section AC, and then drop AC and section A entirely to rank section B, and then BA, BB, BC, and so on, and so forth.

Even more interesting is the fact that I believe a search engine can assign topical value to tiers of a website based on previous tiers. That is, if you run a website about Baseball - and on tier B you discuss teams, and our tier B you discuss years, then a search engine might distinguish parts of your website, based upon the on-page content, as follows:

• Tier A - Baseball
• Tier B - Baseball -> Teams
• Tier C - Baseball -> Teams -> Years

They might continuously carry over the associations from previous tiers into other sub-tiers. That is, they'll recognize baseballs association on tier C even if the tier lacks proper on-site descriptions for the keyword.

So where am I going with this?

Let's say you would like to have Webpage F rank for a "buy baseball cards" query space. It might contain a list of links categorically segmenting your card sales by team. I wouldn't worry so much about optimizing the subsection pages of section F – which would be FA, FB, or FC - for "buy baseball cards," but optimizing them for teams, such as the Cardinals. I would theorize the engines would make associations and conclude that the subset pages of Webpage F (FA, FB, FC, etc) would be something along the lines of "buy cardinals baseball cards."

Mapping websites in this fashion does not come easily and the larger the website, the more complex the process of architecture optimization becomes. Unfortunately, the larger the website is - the more you should be concerned with optimizing its architecture.