Looking For A Few Good SharePointers

It's that time of year again - no, I'm not talking about jolly 'ol Saint Nick, annoying crowds at the mall or an XBox in your stocking (hint, hint), this is even better - The eGroup is hiring! If you've got SharePoint development experience and you're itching to work with a dynamic team of fellow SharePoint-crazy codeslingers, mosey on over to our job postings and see if you fit the bill. No, you won't a free Tesla Roadster and you'll have to bring your own Aeron, but you will get to work on exciting client projects using the latest and greatest Microsoft technologies. Give yourself a Christmas present and tell the boss man you're movin' on up!

UPDATE: UK readers take note - we'll be hiring on your side of the pond in the first quarter of '07. We'll be looking for both experienced and junior SharePoint developers so send in your CV if you'll be in the market for a new gig after the first of the year. In fact, if you're in and about London, drop by the SharePoint UK User Group Meet Up on December 14th (you are a memeber, aren't you?) and introduce yourself (I'll be the only 6'5" Texan in the pub - pretty easy bloke to pick out of a crowd).

SPSiteDataQuery vs. GetListItems

Mart Muller over at TamTam posted his thoughts about using the new SPSiteDataQuery object in MOSS 2007 to retrieve items in a document library without tripping over the annoying folder limitations of the much-maligned SPQuery class. This is handy method for getting list data via the object model but it also has some limitations and 'gotchas' to be aware of.

First, when retrieving list data, anecdotal tests have shown (and our experience has proven this to be absolutely true) that the object model is best suited for working with smaller data sets within web parts or controls in small to medium traffic environments (~10,000 users). Why is this? Nobody seems to know for sure but I suspect that it has to do with the overhead related to repeatedly calling the core DLL's that are also performing a number of other concurrent tasks (but I could very well be wrong; the cause may be up for debate but the results speak for themselves).

Second, as Maurice Prather and Bil Simser have pointed out here, here and here, all those SPSite and SPWeb objects you're throwing around in your code don't dispose of themselves - you have to handle that yourself (you are disposing of your garbage, aren't you?). Those little guys riding round on the back of the Garbage Collector truck never see those objects so they don't throw 'em on the rubbish pile - leave a few thousand of those things open and your application pool will come to a screeching halt.

Third, you have to keep a careful eye on that ViewFields element (and don't forget to write your query correctly). A blank value will return all the fields that the records have in common but not any fields that have blank or null values - these are omitted from the results. This is important when you're looking for a particular field only to discover it's not there at all - crash goes your code. Pass in a null value and you get the default view (such as AllItems.aspx) which may not contain any of the fields you're looking for. You have to specify ALL of the fields that you want to appear in the results. And, just to keep you on your toes, make sure to use the DisplayName value of the column in each < FieldRef > node and not the Name value (they are often quite different, especially in custom lists).

Finally, remember that object model methods of this type are always subject to the annoying but ever-present impersonation issue. Call SPSiteDataQuery in the wrong user context and you won't get very far. RevertToSelf, anyone?

Personally, I prefer to use the web services GetListItems() method to get items from a list because a) it's faster (I know, you don't believe me, but it's true - someone somewhere did a performance comparison once but I can't seem to find it again. UPDATE: Found it here but there aren't any hard metrics in the post. UPDATE II: Scroll to the bottom of this post for some very basic metrics), b) I have more granular control over user credentials (I can use an admin account to get the items then check the user perms to verify the access level - no impersonation required), c) the data is returned as an XMLNode object, and d) there's no garbage to worry about - once the TCP connection is closed the memory is released. Most importantly, the web services scale well, especially in shared services environments where you may have one portal context accessing list data from another portal context. That being said, I will probably use the SPSiteDataQuery object in one-off web parts where scalability and performance are not an issue because the code is short and sweet - which is an advantage unto itself, now isn't it?

New Article on Custom Master Pages

I get a lot of questions from readers and clients regarding deployment of custom master pages in MOSS 2007. There are several ways to tackle this problem and here's a very good place to start - Heather Solomon's new article on How to Create and Store Master Pages on the Server for Use with Site Collections. Heather is one of my all-time favorite MVP's (I swear someday I'm not going to be neck-deep in projects during a SharePoint conference and we'll actually meet in person) and if you're not reading her blog on a regular basis then you're missing out. Go check it out and get those custom master pages going.

To < Query > Or Not To < Query >, That Is The CAML Question

If you do any development using SharePoint's web services you've undoubtedly run across the evil CAML beast rearing it's ugly head to devour your nicely structured markup. For instance, if you invoke GetListItems() to retrieve the column values of each item in a list, you have the option of passing in a CAML query to return specific records or a blank value to get the whole enchilada. The SDK is quite helpful, as it documents how this specific query should look. If, for example, you want a particular record with the title "Yellow", provide the following CAML:

<Query><Where><Eq><FieldRef name="Title"><Value Type="Text">Yellow</Value></Eq></Where></Query>

Simple, no? Well, not quite so simple. Try as you might, this query will never return just those items whose Title column is equal to "Yellow". Instead, the web service will happily ignore your fancy little CAML fragment and give you all the records in the list. Alas, no amount of cursing, head beating, keyboard throwing or foot stomping will make it otherwise. I assure you, I have tried all of those methods, and many more, to no avail. What DOES work is simply removing the <Query> and </Query> tags, like so:

<Where><Eq><FieldRef name="Title"><Value Type="Text">Yellow</Value></Eq></Where>

What's that you say? That's not how the SDK says it should be done? You spent hours searching the newsgroups and nobody once mentioned this little bit of CAML trickery? I feel your pain, my friend, believe me, I do. Were it not for my innate stubbornness and determination to beat SharePoint at it's own game, I would have given up on this vexing problem many moons ago. What can I tell you - CAML is a mysterious creature and we know not it's wild and wily ways. Now go ye forth and slay that noxious Query fiend - or at least stab at it until the next time it bites you on the rear!

MOSS2007 Beta2TR Trials and Tribulations

Today was my lucky day to install a new MOSS2007 Beta2TR virtual machine from scratch. Oh, joy. Has anyone anywhere at any time so far managed to do this without uttering a single curse of frustration? I seriously doubt it. So here are the lessons I learned on today's adventure to MOSS-land:

• If you plan on using a unique admin account (other than 'Administrator') for database access and application pools (such as 'SPSAdministrator') be sure that you are logged in as that account throughout the entire installation, upgrade and configuration process. Otherwise, you'll get all kinds of strange errors during installation and configuration. Oh, and don't forget to add that account to SQL before starting the process with full admin rights (or at least 'dbcreator' and 'securityadmin').
• The slipstream process ain't all it's cracked up to be. I tried three times to slipstream the install, which worked through the installation process (assuming you sorted out the somewhat misleading upgrade instructions), but it failed to create the central admin application upon configuration all three times. What is not quite so obvious is that you need both MOSS and WSS TR's to do the update - just MOSS won't work. This is confusing as you don't need have to install WSS before MOSS the first time around. Just remember to a) copy all the files from the Beta 2 cd image into a local directory, then b) extract the update files for MOSS B2TR and WSS B2TR into that directory's \Update folder.
• Central Administration will not work from a clean slipstream install (at least not for me it wouldn't and I was working from a fresh Win2k3 install). The configuration wizard runs and the central admin app is created but you can't access it - just an annoying 404 error every time. You must first install Beta 2, then run the configuration wizard (which creates the central admin app correctly) THEN run the WSS B2TR followed by the MOSS B2TR.

Finally, after much gnashing of teeth and flailing of limbs, I have a clean MOSS2007 image to work from. Now it's on to custom site definitions which, based on the install experience, promise to be a all-out man vs. machine war. Stay tuned for reports from the battlefield.

Business Portals vs. Line of Business Applications

A primary focus of our business is building custom SharePoint-based applications for our clients.  Naturally, being SharePoint gearheads, we believe that everything under the sun can somehow be made to work in SharePoint (and we're not often wrong) but we run into a great deal of confusion when presenting clients with various customization options.  One of the most difficult ideas to grasp is the separation between a business portal (BP) and a line of business application (LOB).  Which is which and where/how do you use each one?

To begin with, a business portal is defined as any application or set of applications (and the default SharePoint experience is really a set of integrated applications) that share a common brand or visual presentation and is usually delivered as a packaged installation (as opposed to a fixed location extranet or intranet) to customers, partners, or other constituents.  An LOB, on the other hand, is a distinct application that is deployed within an existing inter/intra/extranet implementation. 

On a more technical level, a customized set of WSS site definitions, web parts, and templates that together combine to form a branded collaboration application for, say, the real estate market, would be a business portal.  On the other hand, a set of web parts, lists, roll-ups, controls, or other programmatic elements, delivered individually or packaged together, and installed within an existing company intranet for tracking sales and orders, would be an line of business application. 

Just to muddy up the waters a bit, it should be pointed out that the two are not mutually exclusive - an LOB can be deployed within a business portal and a business portal can be created to serve internal as opposed to external customers (an IT help desk solution for a company with multiple, independent locations comes to mind).  Many organizations start creating LOB's soon after they deploy a new portal solution without realizing that's what they are doing, as they roll out custom web parts that roll-up, consolidate, or provide access to various sources of company data.

Business portals can be a very powerful tool for organizations that need to deliver rich web-enabled applications on a limited budget.  WSS provides a comprehensive, flexible, and customizable (and let's not forget FREE!) platform for creating dynamic web applications.  The best thing about BP's is that the framework is already in place - navigation, data storage (via lists), security, browser-based site management - so developers can focus on customizing and extending the framework without reinventing the wheel at every turn.  With the introduction of SharePoint 2007 and the improved customization options, along with built-in workflow and item-level security, forms-based authentication, and the business data catalog, expect a dramatic upsurge in the number of business portals being created, many of which will be offered as stand-alone applications (hint: expect to see several from us before the end of the year).      

eGroup List Viewer Web Part

Whenever I begin a new portal design engagement, one of the most difficult concepts to communicate to the client's design/architecture group is 'Content-Driven Design'. The theory is simple - create an architecture and taxonomy around the information that is most important within your organization. What makes it difficult is the almost Pavlovian instinct to organize content based on company structure - breaking away from the org chart sometimes requires superhuman effort.

Fortunately, once the initial shock wears off, the light goes on and heads start nodding around the conference table. It doesn't take long for people who couldn't pronounce 'taxonomy' the week before to suddenly become content design experts. This only makes sense, as they already have a picture in their minds of how their company's information should be organized, they just need a push in the right direction to translate those thoughts into a good architectural design.

One of the most critical elements to a good content-driven architecture is the concept of "Author Once, Publish Everywhere". In this model, information is created and managed in the most granular context applicable to it's function but published in the context where it is most often consumed. For example, an HR policy would most likely be created and stored in an HR team site, somewhere within the overall departmental hierarchy. This allows team members to collaborate effectively on documents that pertain to their job function while permitting the greatest degree of security and oversight. Users, on the other hand, shouldn't be poking around in the HR team site, they should be browsing a portal area that is generally classified to contain that type of information (such as 'Employee Resources' or 'New Hire Information').

Unfortunately, this design methodology reveals an inherent weakness in SharePoint - the inability to distribute list content throughout the portal. In the SharePoint structure, lists can only be viewed in the context in which they were created (in our example, the HR documents could only be viewed within the HR team site). SharePoint does not ship with out-of-the-box capabilities to create list items in one area/site and view them in another.

To meet this requirement, we created a web part which allows list content in any area or site to be viewed in another area or site by users with sufficient permissions to view items in the source list (they must at least be a 'Reader'). We've used it in numerous client engagements and it is a critical component in our Content-Driven Design methodology. Just add the web part to a page, give it the portal/site url, list name, and view, set the display options (toolbars and column titles), and voila - list content for everyone, everywhere!

The web part can be downloaded here. The zip file contains a ReadMe document with instructions on installing the CAB file (in case you haven't done it before) along with information on deploying it to a portal/site page and configuring the various options.

As always, post any questions, comments, or suggestions to this post.

Enjoy!

eGroup SharePoint Utilities

I am a big proponent of leveraging SharePoint to deliver enterprise line-of-business applications (LOB's). Much of the custom development we do at The eGroup involves the use of SharePoint lists as data repositories for dashboards, workflows, roll-ups and, of course, LOB's. Working extensively with SharePoint lists brings introduces its own set of challenges - coding against the web services, creating and maintaining a large number of custom views, managing security and migration of list structure/content from development to staging and production.

In the course of developing these applications we have created a set of tools to aid us in our efforts. To save our fellow SharePoint developers some of the headaches we've encountered, and to encourage other developers to contribute their time-saving apps, we'll be releasing various applications and utilities from our toolkit in the coming weeks and months. The first utility, the SharePoint List XML Viewer, received an enthusiastic response and generated a fair number of downloads. I hope it is serving everyone well and we will continue to improve it as we receive feedback from the SharePoint community.

The second tool to be released, the SharePoint List Creator, is a companion utility to the List XML Viewer. Using the XML output gathered from the XML Viewer, it provides developers with the ability to define custom lists in XML and easily deploy them to multiple sites/areas. This can be quite a time saver when you have a large number of heavily customized lists that must be moved to different environments, especially when each portal or site collection exists in disparate server farms and/or separate domains.

To use the SharePoint List Creator you'll need good working knowledge of XML and some CAML experience. Creating a new list requires a source XML file that contains the properties, fields, and views that will comprise the list. A sample file, DocumentLibrary.xml, has been provided in the \XML directory to give you a basic template to work from (remember to store all your custom list templates in this same directory). If you haven't done much CAML or manual list creation, a good first step would be to create a custom list in the SharePoint GUI then view the list structure with the List XML Viewer. Compare the output to the sample xml template and it won't take long to figure out how to create just about any list type from scratch. We'll release more samples in the future to aid new developers and in response to comments/requests.

To use the SharePoint List Creator, first download the eGroup SharePoint Utilities Windows Installer package from our web site. This package contains both the SharePoint List Creator and the SharePoint List XML Viewer. Next, install the MSI on your development machine and launch it from the program menu shortcut. Provide the site/area URL, list name, type, source XML document, and authentication credentials, and submit the changes. The tabs in the utility will display the source XML, new list XML being submitted, the results from the list updates, and the results from the view modifications. You'll also find any errors encountered under the appropriate tab (i.e. if view creation fails, check the "View Results" tab).

We'll update the Utilities suite with new apps as we convert or create them (many are ASP.NET applications that require migration to Windows Forms prior to deployment). As always, please post your questions, suggestions, feature requests, gripes and complaints as comments to the post. Feedback is the only method we have to gauge the community reaction to our efforts. I hope you find the tools useful. Enjoy!

New Version of SharePoint XML List Viewer

I received some good feedback on the XML List Viewer utility and have posted a new version with the following modifications:

* Clear text entries hidden in password field
* XML displayed in structured view (similar to IE functionality)
* XML can be saved to local file (Right-click within XML field, choose 'Save')
* XML can be printed (Right-click within XML field, choose 'Print')
* XML can be searched for a specified string (right-click, 'Search')

Download the new version (1.1) here.

Please post any additional feature suggestions as comments to this post.

Enjoy!

UPDATE: Downloads from the earlier post had problems with the .NET 1.0 version of the tree view DLL. I have posted an updated version (use the same link above) that includes a recompiled .NET 1.1 version XmlTreeView source code within the executable (thanks to Thomas Siepe for making the source available on The Code Project).

SharePoint List XML Viewer Utility

Many of us are rejoicing over the new web publishing features in MOSS2007 and the ability to create corporate web sites using SharePoint. Unfortunately, many enterprises are not planning to upgrade to 2007 for quite some time, and those that use SharePoint extensively are looking for ways to derive additional value from their portal investment. One of the best ways to do this is to use SharePoint 2003 as a basic content management system for a public web site (at least until a full 2007 migration can be completed).

I just finished a project where an existing - and very expensive - CMS application (I won't name names but it's not MCMS) is being phased out in favor of using SharePoint lists as the primary method for managing content on a standard ASP/ASP.NET public web site. When you think about, this makes perfect sense - SharePoint provides a form-based web interface for handling content submissions, security is built into the platform, content approvals and basic workflow are easy to implement, and every list in a portal/site is accessible via web services (for more information on using SharePoint as part of a basic CMS application, email me directly).

While working on this project I discovered that working with the SharePoint web services is no picnic; not only do you have to know all their little quirks (like the viewName argument requiring a GUID instead of the display name of the view) but you also have to make allowances in your code for all the funky things SharePoint does behind the scenes with the list XML. For example, when you name a field 'Field1', the list details show the field as just that - Field1 - but when you try to retrieve the list data using the Lists web service, you must refer to the field as 'ows_Field1'. And then there's the ever-popular trick of replacing all spaces in field names with '_x0020_'. Naturally, there are no tools in SharePoint that allow you to retrieve this information, invaluable though it might be. It's enough to drive you mad.

To maintain my sanity and bring some order to web service programming, I created a small utility to retrieve the XML for any list via the SharePoint web services. The SharePoint List XML Viewer is a small Windows app that programmers can use to retrieve the XML output for any list or list items (including any default or custom views) to aid them in building web service applications. You can download it here. Just unzip the file and run the executable. Give it the site/area url, list name, and view name, supply the necessary login credentials, pick which type of XML you want (list data or list items) and - voila - instant XML that can be cut-and-pasted into your favorite text editor.

The SharePoint List XML Viewer Utility isn't anything fancy - it's just a basic utility for developers - but it saved me countless hours programming against SharePoint's web services and I hope it helps you, too. Enjoy!

(Post comments to this post if you have any issues installing and using the tool)

BTW, I recommend using the application along with the CAML Query Builder and Execution Tool from the good folks at U2U. Sooner or later you'll need to execute a query in a web service call and the CAML Builder is the best thing going.