Optimizing External Site Crawls

After launching SharePoint Blogsearch, I discovered (with some help from Greg) that the SharePointPSSearch (SPSSearch) service needs a few tweaks to work well on external web sites. After doing some digging around in the documentation, looking into the packets with a protocol analyzer, and generally scratching my head in confusion, I learned a few things:

1. SPSSearch does not always honor robots.txt files (this is a text file placed in the root directory of a web site that tells crawlers how to behave). Yes, the documentation says it does, and you can modify the id string in the registry, but it doesn't always seem to work. I'm still trying to come up with an answer to this one.

2. By default, the crawler will request as many documents from the target site as it can fit into the available threads or until it starts receiving TCP errors; in other words, it will hammer an external site into submission. Fortunately, you can control this errant behavior. Go to SharePoint Central Site Administration Manage Search Settings. In the 'Site Hit Frequency Rules' section, click on 'Manage Site Hit Frequency Rules'. Click 'Add Update Rule' on the toolbar. In the Site Name field, enter "*" for all web sites (you can also set rules by explicit name, domain, etc.). Next, click the 'Limit number of documents requested simultaneously' radio button and enter a small number (minimum is 1, max is 999, I used 5) in the Number of Documents field. This will significantly reduce the load on target servers.

3. Incremental updates are SUPPOSED to ignore content that has not been changed, crawling only those docs that have changed. In reality, this is not the case. I noticed that all documents were being processed even though the content was static. I tested a full update and incremental update on a static site and the exact same load was generated (packet count, bytes, etc.) in both a full and incremental update. This could be a bug or it could be some hidden registry setting somewhere that I haven't found yet. Any ideas would be appreciated.

4. Adaptive updates only appear to work on SPS/WSS sites. According to the docs, adaptive updates make an educated guess, based on historical patterns, as to what content may have changed on a site. In theory, this should greatly reduce the load on a crawled site and it may very well work that way in SPS/WSS. It doesn't seem to have any effect on other types of sites but my test configuration was limited so I may not have all the data. Again, if anyone has any ideas, please share.

More on this topic as I continue to tweak the settings.

Update: If you want to learn more about search optimization, here is a KB article to get you started.

Update 2: Here's another tip. When creating an inclusion rule for a subdirectory on a site the default behavior is to also include the parent site. For example, a new inclusion rule for http://www.theegroup.net/blogs/ would create two entries - one for the full path and one for the parent http://www.theegroup.net. This means that any links to other URL's on the same site would be processed which greatly expands the scope of the crawl. To restrict this behavior, change the parent rule to an exclusion and leave the child URL as an inclusion. This will restrict the crawler to links under the child URL.

Custom Lists Article

Here's a fantastic article from Heather on adding custom lists to a site definition.

Useful Shortcuts

I love it when I'm just wandering around and find something truly useful (OK, I wasn't wandering - I used SharePoint Blogsearch to find one thing and stumbled across another). Here's a post from Todd Bleeker regarding javascript links that toggle page modes in SPS and WSS. I already new about appending ?toolpaneview=2 to the URL but this is even easier - and he explains what all the options are. Very cool. I had no idea you could do this with IE favorites.

Also, here's a great post from Serge van den Oever on FrontPage and Data View Web Parts. Very thorough and a must read.

Sharepoint Resource List

While responding to a post from Heather on the proper uses of FrontPage with SPS 2003, I discovered her list of SharePoint resources. Very handy. Thanks, Heather!

SPS Report Utility

Ryan has posted a cool new utility for gathering SPS and system info. Get it here.

TechEd 2005 SharePoint Webcasts Online

The webcasts from TechEd 2005 are now online. Get them here.

SharePoint Blogsearch

While Feedster and Technorati do a fine job searching general blog content we SharePointers are a bit out in the cold when it comes to searching posts from our fellow bloggers - not all posts have the keyword "sharepoint" and there is no way to set up a group of blogs to search. What are we to do?

Well, thanks to the much-maligned-but-ever-so-useful SPS search, we have an answer. I've taken my blogroll and created content sources and a new search scope on my website. Just select the 'SharePoint Blogs' scope and go to it. Incremental updates run every 4 hours - I'll adjust it as necessary depending on search results and bandwidth consumption.

If you've got a blog you want added to the list, email me and I'll add it. Also, if you want to access the indexes (or any web services, for that matter) from your site and need a login account, just let me know.

Happy searching!

The SharePoint Vision

Heather kicked-off a mini-tsunami with her post on the SharePoint vision. Andrew then jumped in with a few more thoughts (here and here). Bil's got a few ideas in his response to Heather's article (see her original post).

This is a good discussion. Anytime we stop writing code and start talking about the user is a step in the right direction. Heather has some very good points but one in particular is striking enough for me to add my two cents (I think we're up to a buck by now with everyone's input - or about 50 pence for my friends across the pond). She wonders why there aren't more general use web parts available to the SharePoint community. It's a good question with some uncomfortable answers:

1) Skilled SharePoint developers are few and far between. Sure, there are lots of people who have worked with SharePoint in some capacity but very few who know what they are doing and are actually good at it. This is by far the most common complaint I hear from clients and one that causes me a great deal of grief when trying to find adequate resources for a project. We NEED more and better training. We NEED SharePoint certifications (Are you listening Microsoft? Four years on the market and no cert track? Are you kidding me?).

2) The object model is preposterous. Yes, I know SPS was cobbled together quickly to meet market demands and WSS is just not that mature (props to MS for delviering a good product under those conditions despite what Mike Drip may think); however, they've got to clean up the OM in the next release. I've seen firsthand how it frustrates many .NET programmers who could otherwise be writing good code instead of wrestling with SharePoint nuances. Simplify and productivity will improve.

3) Due to #1 and #2 commercial web parts are very expensive. Solve these issues and the number of web parts will increase dramatically with a corresponding drop in prices. Developers should always be encouraged to profit from their efforts, so let's not expend a bunch of energy arguing about open source, community, freeware and so on. The profit motive is the most effective, so let's continue to encourage commercial development and take advantage of basic supply and demand principles - the more providers the lower the price. In fact, I'd like to see MS encouraging the Micro-ISV concept with new partner programs and support.

4) No centralized web part repository. As Bil astutely points out, we have to scour the web to find web parts like we're caught in some gigantic SharePoint treasure hunt. We need a better way to organize, categorize, rank, and share feedback on every web part out there. MS has taken a good first step with the component directory but it's just cut-and-paste marketing speak - no objective reviews or comments. I'm all for Bil's SharePointForge idea.

5) Corporate intellectual property restrictions. Many organizations don't want the result of their employee's efforts released into the wild without compensation. I see this happen more often than you might think. Right or wrong, and it's often a mix of both, they worry that competitors will benefit from their investment or that employees will spend their time supporting people other than their own internal users. Not to mention the most significant concern of all - liability. All it takes is one lawsuit to stop the flow of good code altogether. This is mostly a corporate culture issue but could be solved in some instances by better communication with management and pre-authorization from legal and marketing.

Those are just a few of my thoughts. I encourage the community to keep this discussion fresh and in the fore.

UPDATE I: While I'm on this soapbox, allow me to put in a plea to the developer community for standardization in product packaging. It's not that hard to compile a CAB file - don't make users jump through hoops to implement your code. Perhaps this should be a requirement in whatever centralized directory scheme is (hopefully) implemented - poor packaging equals poor ranking and fewer downloads. Some sort of incentive system for highets rankings and most downloads would be great as well.

UPDATE II: For anyone that is interested I would be willing to host the directory on my SharePoint server (c'mon, it has to be in SharePoint, doesn't it?). I don't think I can handle the storage and bandwidth for all the downloads but I would be more than happy to host and maintain the code. Any takers?

Site Navigator v2.0 Beta

Advis has just released a beta version of Site Navigator v2.0. This is by far my favorite site navigation web part. Download the free beta release now (registration required) and post some feedback to the Advis team - they're a crack bunch who really want to hear about any product issues and/or feature requests.

Sharepoint and CMS Update

Not to inflate my own stock but I've been predicting this for months. Look for future announcements regarding Microsoft's positioning of SharePoint as the recommended back-end storage system for Office, an "Office Server" package of some sort, and transparent integration between Visual Studio Team System and SharePoint (including a port of SourceSafe to SharePoint).

Hey, they're just predictions, but I've been on the mark so far. Keep an eye out and let me know where I've gone wrong (or right).