Actually, now is a really good time to have this conversation because with the work I am doing on the Exchange Web Service code, I have just finished revamping my internal infrastructure to support the equipment and software I need to do the job. So this is going to be a two-part article. In this part, I'll tell you about the software development components that I use. In the next part, I'll tell you about the infrastructure components. The problem is that you need to understand some of the details of why, so I'm going to start with a little background.

Background

There are a few basic facts about me that effect everything else:

1. I am not rich. I have to rely on the most cost-efficient solutions I can. That means that I prefer open source/free software where it makes sense.
2. I am not cheap. I recognize that building software that is commercial grade requires a serious commitment to quality. That means a serious commitment to the development process, which, in turn, requires things like solid designs, automated builds and testing, and extremely good discipline.
3. I am the worst perfectionist. A gentleman I used to work for back in South Africa, Jan Fernhout, used to tell me that I would way over-engineer things because I wanted a perfect design. That is true. I have to be very disciplined about remembering that I need to deliver something.
4. I hate mundane work. If at all possible, someone else is going to do the work for me, and I there is no way that I am going to manually do something that can be completely automated. If you make me do it, I will bleat like a sheep until you fix the problem – just ask some of my co-workers.

There are also a few facts about the development work that I do that affect everything, too:

1. I believe that Windows is for the desktop or as a departmental product and Unix is what the enterprise needs. No one has proven me wrong on this. For every example of a large Windows product installation, I can point you at hundreds of more successful, more cost-efficient, more stable, less administered Unix installations that just run.
2. I do a lot of work with both Unix and Windows.
3. Java is the most platform-independent language for back-end work.
4. .NET is the richest GUI experience you can have outside the browser on the Windows platform.
5. OpenEdge is a fact of my life. I have used it more than any other development platform. I have made a good living off it, and I love the product, warts (many of them) and all. OpenEdge is kinda like an old pair of shoes that fits your feet a certain way and you're always most comfortable wearing them, no matter how ugly they are.

Machines and Operating Systems

My desktop machine – and primary development workstation – is a Windows 7-based HP Pavilion Entertainment 64-bit laptop with 6GB of RAM and 450GB of disk space. It runs nothing but Windows, and I have a ton of software on it.

My primary server machine is a  Dell PowerEdge T710 with 2 4-Core Intel Xeon processors, 16GB of RAM and 2 500GB drives. It runs VMWare ESXi (see screenshot at right) with 20 different virtual machines on it, some constantly running and some not. I'll tell you more about this in the next post on infrastructure. There are 6 VMs, though, that are important to this discussion:

• A Windows Server 2008R2 primary domain controller (pdc);
• A Windows Server 2008R2 build server (winbuild);
• A Windows 7 32-bit build machine (clientbuild);
• A CentOS-64 development server that hosts a whole lot of stuff (develop);
• A CentOS-64 build server (build64); and
• A CentOS-32 build server (build32) .

I then have 2 other machines that are both older machines that I use for various things. I'll talk about these in my infrastructure post, too.

Components of a Development Environment

As I said, I'm the worst perfectionist. So development starts with an idea, and ends when the code has been packaged, deployed, and installed on the end-user's machine, and they have completed user acceptance testing. In other words, it not only spans the entire Enterprise Architecture stack, but it also spans the Software Development Life-Cycle (SDLC). 

Now as much as I said I am not rich, there are two very expensive tool-sets that I have to buy:

1. Visual Studio Premium with MSDN. This one sticks in my throat every time I pay for it. The only reason I need the Premium subscription is for the Windows Server and Microsoft Exchange Server licenses. The more than $5500 that this product costs is very hard to justify, and the $2300 a year charge for renewing the subscription is as bad as paying tax to the IRS. I am a firm believer that development tools should be free. If you want me to develop products for your platform, give me the tools I need to do it.
2. PSDN Professional. Although I also believe that Progress should provide its tools for free, I am less resentful about it, probably because I used to work for the company. I still think the $3,000 that we pay for PSDN is excessive, but there you have it. My guess is there would be more people developing Progress-based applications if the development tools were given away free.

I need the MSDN license for the operating system and server licenses that I am using for Exchange Server. For my non-Microsoft-based development, I rely heavily on the Java and LAMP (Linux-Apache-MySQL-PHP) stack. Because I have some experience with PHP, I try as far as possible to use Open Source tools that are based on PHP rather than Perl. I don't know Perl and I don't particularly want to spend the time on it.

So starting from concept, here are the tools that I use to get my stuff done:

•  Sparx System's Enterprise Architect (EA) for all my modeling (not just diagrams, models). The product cost me about $300 for the original license, which included the MDG technology for Eclipse integration, and I use it as a design tool from the conception of an idea all the way through delivery. It contains the business process models, the requirements, and all the UML diagrams and test cases. I am not a fan of model-to-code driven development. I think the fact that we have been looking at the equivalent of CASE tools for the last 20 years shows that there will always be those cases where the code cannot be automatically generated for you. Two of EA's strongest suites are the traceability it provides from requirements to test case, and the incredibly powerful document generation facility. I literally generate all my documents from Enterprise Architect.
• Collabnet Subversion is used as my Source Code Control System. Everything goes into Subversion, including my models, source code, document artifacts, and even my research code. Subversion runs on my development server (develop) on CentOS. Subversion has clients for all kinds of platforms, so I have AnkhSVN plugged into Visual Studio to control my .NET code, I have Subclipse plugged into Eclipse for all my Java code, and I use TortoiseSVN at the operating system level to control documents and stuff that are outside of either development environment.
• Mantis is used as my issue tracking system. There is an Eclipse plugin for Mantis so I can get at it directly from my project workspaces in Eclipse.
• Hudson CI is used as my automated build and test environment. It runs on develop, but starts builds on build64, build32, winbuild, and clientbuild. I use both Apache Ant and Apache Maven to build Java code, and I use PCT to build OpenEdge code in Hudson. I use Nant to build .NET code.
• Unit testing is the scope of JUnit and NUnit although I am vehemently opposed to test-driven development. I think it is flawed and creates really lousy software.
• Eclipse is my Integrated Development Environment of choice. I have at least two installations of Eclipse at any one time. One is for non-Enterprise development (stuff that does not require Java EE) and the other is for Enterprise development (Java EE). The Eclipse screenshot at the top of this post shows my working environment for non-JavaEE work. From the screenshot, you will notice that I have OpenEdge running inside the Eclipse environmnent (Database Explorer is at the bottom, and there is an OpenEdge Editor open above it. You can also see an Enterprise Architect model in the right-hand pane, and my Hudson build status is in the left-hand pane. In the Eclipse screenshot at the bottom of this post that shows my working environment for JavaEE work, you will see the Mylyn task list at the bottom on the left, with the Mantis task repository open in the pane on the bottom right and a Mantis task open on the right. All of the tools that I have highlighted are integrated right into my Eclipse IDE in both environments. 
• Visual Studio is used only for development work that doesn't work in Eclipse. So most .NET/Windows-based development is done in Visual Studio. I have both Visual Studio 2008 and 2010 installed. 

All the tools I have discussed so far have plugins for Eclipse and, as the screenshots show, that is the configuration in which I use them.

A couple of other products that I use extensively that are not really integrated into the development environment per se, are:

• Beyond Compare is one of those tools that I just cannot imagine not having. It is a tool for doing advanced 3-way compares of files and it is, without reservation, the best diff tool that I know. It runs on both Windows and Linux, and I use it on both.
• WinZip is another tool that I use so much, I cannot imagine being without it. It is used for creating zip archives, but it can just as easily read and understand Linux tarballs and .jar and .war files.
• Nero is an outstanding tool for burning media, especially CD or DVD burning.

Most of my publishing stuff is done with either WordPress or Drupal. Like most of the other stuff I use that is web-based, these two are both PHP-based solutions. PHP is not a hard language to learn and it is very good at getting pretty sophisticated web apps up and running relatively quickly. WordPress and Drupal both have extension APIs that make it possible to build pretty sophisticated plugins for both in PHP – something I have done reasonably effectively.

Conclusion

Most of the tools I have mentioned above are freeware or open source or really, really cost-effective, and the good thing about them is that they lead you to build much better quality code than would otherwise be the case. If you are not looking at tools like these, I would highly recommend giving them a try.

In my first post, I told you about the background story – I need to enable an OpenEdge CRM application to create, modify and delete calendar and task items in Microsoft Exchange. I also need Exchange to let me know any time a calendar or task item is changed so that I can update the OpenEdge database accordingly. Simple use cases.

When I left off last week, my next step was to get Exchange subscriptions working, and, boy, what a trip that has been.

Types of Subscriptions

There are two types of subscription models available with Exchange:

1. Pull Subscriptions – You set up a subscription with Exchange and it gives you an indicator (called a Watermark) that keeps track of what events you have already received. You pole the server at an interval and it gives you a response with all events that have happened since that last Watermark.
2. Push Subscriptions – You set up a subscription with Exchange and tell it to call you back on another WebService every time something happens. The WebService has to acknowledge receipt of the message in order to keep the subscription alive. The message you get contains a list of items that have been affected since the last time a message was acknowledged.

I'm not interested in polling the server so I have chosen to go with a Push Subscription. Much of the information that follows applies to both subscriptions, but the Push Subscription differs in that you do not poll the server for the changes. 

Setting up a Push Subscription

To set up a Push Subscription, you send a SOAP message to EWS, authenticating against the Active Directory as you would with any other message to EWS. The message contains a list of folders that you want to know about, the list of event types that you want to be notified of, the URL that you should be called back on, and a StatusFrequency node that tells it how often to poll the URL – more about this later. 

You can subscribe to virtually any update on virtually any folder in a mailbox, including the calendar, tasks, contacts, and mail folders.

The diagram to the left shows the architecture that I finally ended up with. It is a little more complicated than it would be if OpenEdge were not in the picture, but it serves to explain the process, nonetheless. The numbers in parenthesis below refer to the numbers in the diagram.

OpenEdge initiates the subscription by sending a message (1.1) to a Java WebService that is deployed on a Glassfish Application Server. The WebService forwards that subscription request (1.2) on to the IIS server that is running on the Windows 2008 Server that hosts the Microsoft Exchange Client Access Server (CAS). IIS passes the message onto the EWS API which registers the subscription with the CAS. CAS responds with a message that contains the Subscription ID and the Watermark which is tied to the last event that was sent.

As I mentioned, the message that is sent to the EWS API (1.2) contains the URL for the service that is to receive all event notifications. It also contains the StatusFrequency node.

I set up a servlet on my Glassfish Application Server to receive and respond to the notifications. The following is a sample subscription request message:

<?xml version="1.0" encoding="utf-8"?>
    <soap:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
           xmlns:xsd="http://www.w3.org/2001/XMLSchema"
           xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"
           xmlns="http://schemas.microsoft.com/exchange/services/2006/messages"
           xmlns:t="http://schemas.microsoft.com/exchange/services/2006/types">
        <soap:Body>
            <Subscribe xmlns="http://schemas.microsoft.com/exchange/services/2006/messages"
                    xmlns:t="http://schemas.microsoft.com/exchange/services/2006/types">
                <PushSubscriptionRequest>
                    <t:FolderIds>
                        <t:DistinguishedFolderId Id="calendar"/>
                    </t:FolderIds>
                    <t:EventTypes>
                         <t:EventType>CreatedEvent</t:EventType>
                         <t:EventType>ModifiedEvent</t:EventType>
                         <t:EventType>DeletedEvent</t:EventType>
                    </t:EventTypes>
                    <t:StatusFrequency>3</t:StatusFrequency>
                    <t:URL>http://server.example.com/ewsCalTest/NotificationService</t:URL>
                </PushSubscriptionRequest>
            </Subscribe>
        </soap:Body>
    </soap:Envelope>

The StatusFrequency Node

The StatusFrequency Node tells EWS to send a message every x minutes to verify that the subscription is still active. This serves three purposes:

1. It allows Exchange to figure out if the server that is listening is still around and cancel any dead subscriptions if necessary;
2. It allows the server that is being called to determine that it is still getting messages from the Exchange Server; and
3. It allows the server that is being called to unsubscribe from any future messages.

Exchange needs to carefully manage resources, so canceling any unnecessary subscriptions is an important way to do that. To maintain a subscription, the server that is receiving the notifications has to respond with a SOAP message that includes a SubscriptionStatus node with the value "OK". 

The expected response is as follows:

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope">
  <soap:Body>
    <SendNotificationResult xmlns="http://schemas.microsoft.com/exchange/services/2006/messages">
      <SubscriptionStatus>OK</SubscriptionStatus>
    </SendNotificationResult>
  </soap:Body>
</soap:Envelope>

If a subscriber does not respond appropriately, EWS goes into retry mode. In other words, if EWS does not get a SOAP message formatted exactly the way it expects, it assumes the server has not received the message. It waits for 30 seconds and tries again. If it still receives no response, it waits for a minute and tries again. EWS will continue retrying, doubling the wait time, until the wait time exceeds the StatusFrequency. So in my case, when I first set this up, I set the StatusFrequency to 3 minutes. It tried to send the notification and waited 30, then 60, then 120 seconds and canceled the subscription because, if it had gone to 240 seconds, it would have exceeded the 3 minutes set as the StatusFrequency. 

Less obvious, though, is the reason behind purposes 2 and 3. Remember that Exchange is only going to send messages when something changes. Well, some users have very quiet mail boxes. They may be out of the office, or it may simply be after hours where nothing is going on. If Exchange does not let the other server know that it is still sending messages, how is that server to know that the subscription is still valid?

The other thing that is important is that there is no way to send a message to Exchange to tell it to cancel a Push Subscription. The only way to cancel the subscription is by responding to a status message with the SubscriptionStatus set to "Unsubscribe".

Receiving Notifications

Now that you understand how to subscribe, the next step is to start receiving messages from EWS. We'll pick up where we left off in the diagram above.

EWS starts off by sending a notification (2.1) to the server. The notification could be one of two things:

1. A notification of events that have taken place on the server; or
2. A status message.

If this is a status message, all you need to do is respond with the OK response (2.2) I discussed above. In fact, you need to respond with the OK message if you want to carry on receiving notifications no matter what, but responding with an OK has an additional effect if you receive a notification of events.

First let's understand a notification of events. An event notification message contains a list of events that have taken place against the items that you are monitoring. It does not contain the details of what changed – just the fact that something did. If you want to know what changed, you have to call the GetItem method of the EWS to get the item and look at its changes. That's what we do in step 2.3.

Each notification includes a PreviousWatermark for the whole message and a Watermark for each event. By keeping track of the watermarks, it is possible to restart the subscription from the last event that was processed. When you acknowledge the event notification with an OK, Exchange moves the subscription to the next watermark automatically.

Once I have processed the event notification and received the event, I am making a Dynamic OpenClient call (2.4) to the OpenEdge AppServer to notify the AppServer of the change. The ABL code on the AppServer can then do what it needs to do to respond to the event.

Performance

Once I got this working, I decided to test it out to see what the performance looked like. Now bear in mind, all of the machines shown in the diagram are different virtual machines on one Dell Poweredge T710 running VMWare 4.0 ESXi. The T710 has 2×4 core processors and 16GB of memory. It has 4 network cards connected to a 1GB switch. The Windows Server is on a separate network card to the other three machines, which are all Centos 64-bit boxes. 

The round trip for establishing the subscription takes an average of 82ms.

I then ran a test harness that generates 100,000 calendar items across 80 mailboxes. It took a little over 2 hours for the test harness to complete. I tracked performance for each of the following pieces:

• The OpenEdge client takes an average of 100 milliseconds to make the call through the Java WebService to EWS to create the appointment.
• From the time I receive the acknowledgement for the appointment creation to the time that the notification has been received (2.1) and acknowledged (2.2) and the details of the notification have been received (2.3) takes an average of 150 milliseconds.
• The call across the Dynamic OpenClient to the AppServer, which is a call to a procedure that has no temp-tables or XML documents, averages 80 milliseconds.

All told, this means that from the time the request for the appointment is initiated to the time the notification of its update is received by the AppServer takes an average of 330 milliseconds – less than half a second.

Now clearly, my environment has very little traffic and I have a lot of horsepower backing it, but it if this is properly tuned and the additional processing that is to be added is not excessive, a one second response time is well within the realms of possibility.

Problems and Challenges

In getting this to work, I ran into a couple of nasty obstacles. 

First, Microsoft's API is really not a good example of a WebServices API. Neither Java nor OpenEdge will parse the WSDLs without manual editing and even if you manage to generate the code you need from the WSDL, it really doesn't work very well. I have been forced to manually code the interaction with the EWS API and that is a lot of work. Having said that, it performs very well and I get to write the XML exactly the way it needs to be for EWS to accept it.

That brings me to the second thing about this API. Microsoft's error reporting is shockingly bad for commercial software. Compared to the kind of error reporting I get from Tomcat, Glassfish, ActiveMQ, and several other Open Source solutions, it's incredible that commercial software can be this badly engineered. It is so hard to figure out what is wrong when something goes bad that you will spend hours on things like malformed or unrecognized XML. EWS simply returns a 500 http error code (Internal Server Error) with absolutely no indication of what caused it, and there is nothing in any of the logs to help you.

Update: Since I wrote this article, I have found (and Kris C. has also pointed this out in the comments) that making a call to the getErrorStream() method on the connection object will provide more detailed information on what failed, and you get an entire SOAP message back that describes details of the 500 error. I therefore eat my words about error reporting.

I still think that there is room for improvement because it is very hard to diagnose exactly which node is out of order if you submit a request where the XML nodes are not in the exact order specified by the EWS API, but it's nowhere near as bad as what my original post probably led you to believe.

There is more information in Part 3 of the series and there is even sample code that demonstrates exception retrieval.

The third thing that really cost me a lot of time is that you cannot call a regular Java or OpenEdge WebService from EWS because it won't look at the WSDL. I was therefore forced to manually code the Java servlet that accepts an http post.

These problems effectively sidelined OpenEdge as a direct endpoint for EWS notifications. The architecture that I am building will definitely rely on all communication with EWS going via Java services.

Where to next?

Now that I have the prototype code working, my next step is to fix up some of the security issues I need to resolve. NTLMv2 authentication remains a real problem for Java in communicating with EWS, and I am looking for a solution around that. I also need to deal with some of the certificate related issues that I have.

I have also done some work on the Exchange Impersonation stuff, so my next blog post will probably talk about this.

Finally, I'm planning a walkthrough of some example code that actually demonstrates the functionality, like I did with the Dynamic OpenClient code late last year. 

A couple of months back, a gentleman who has now become a friend and business partner, Jim Ford of Ford Audio-Visual, came to me and asked me if there was any way to get at all the calendar items in his sales organization's calendars with the intention of integrating it with his Progress OpenEdge CRM system. Jim is using Exchange 2007 for his e-mail and calendaring solutions.

I was aware that Microsoft had released a new API for Exchange in Exchange 2007 called Exchange Web Services (EWS), and so I said that I needed to do a little research on the API, but I was pretty sure that it was possible. Sure enough, MSDN has some documentation of the API and Microsoft is touting it as the replacement for all APIs that communicate with Exchange. Web Services – how hard can it be?

The Production Environment

In the production environment, an OpenEdge client running on a Unix server needs to initiate a WebService call to Exchange to create, update and delete appointments for sales people. These appointments will be created in the user interface for the CRM application.

Sales people then need to be able to change and delete these appointments from their Outlook clients or their Blackberry/iPhones. Any changes to the appointments need to be recorded as history updates in the CRM system, so Exchange needs to notify the OpenEdge CRM system of these changes.

So the updates to appointments need to happen from both directions and the communication is going to happen cross-platform (Unix to Windows and vice-versa).

Initial Idea

My initial plan was to create a prototype that did the two-way communication directly from OpenEdge to EWS. I just wanted to prove that it was possible. The first step was to build an OpenEdge WebServices Client that created a calendar item in the Exchange store. Unfortunately, for various reasons, this is much harder than I thought it would be.

First, the EWS WSDL is not a standard WSDL. EWS is installed as part of the Client Access Server (CAS) component of Exchange. In production, you can have multiple Mailbox Servers and the CAS has to discover which mailbox server services a client. As a result, the URL that is used to actually call EWS is different depending on which Mailbox Server hosts the mailbox. The WSDL therefore does not contain a "service" node and the OpenEdge WSDL parser cannot parse the WSDL. I tried working around that by creating a dummy WSDL with a service node inside it.

The second problem was that I ran into all kinds of problems with the complex type structure that exist in the WSDL. The types and messages live in a separate XSD file and the WSDL parser did not like some of the message definitions as they included optional types.

The last straw was that the connection to EWS is an SSL connection that, by default, wants to use NTLM authentication and I battled for hours to get this to work. I would probably have figured the problem out earlier if I had had some more clear indications from OpenEdge of exactly what the problem was, but I finally decided to go and try the connection from Java.

The Bible

Before I started on the Java example, I decided that I should probably test that I could even make this happen in .NET. Even that is no small feat. The examples that come with the Exchange SDK are about as useful as an udder on a bull, and I eventually decided to bite the bullet and look for a book on the subject.

I normally avoid Microsoft Press books because, more often than not, they simply regurgitate the standard Microsoft examples and contribute very little additional value. That is not the case with Inside Microsoft Exchange 2007 Web Services. Authors David Sterling, Benjamin Spain, Michael Mainer, Mark Taylor, and Huw Upshall have done an outstanding job of creating a book that really gets to the heart of the technology. They are all members of the Exchange development team, and it shows. The book provides enough background information that it is essential reading for anyone trying to do this stuff.

Java to EWS

Armed with my new reference, I now set about building a Java client that could create an appointment in Exchange, figuring that I could expose the Java client as a Web Service that OpenEdge could consume.

Now, before you throw your toys out your crib and tell me that that is a very expensive way to do things from a performance point of view, let me just clarify that I would have ended up going down this road eventually, anyway. Ultimately, the solution will be exposed as a service on an ESB, and that ESB is likely to be based on a JMS MOM, like Apache ActiveMQ, or SonicMQ. So this is not a wasted exercise. The reason that I need this on an ESB is that the events that flow in each direction need to be available to other services, too.

Java to EWS was not a lot easier to get working. I still had to work around the WSDL not having a "service" node issue, but Java gives you finer-grained control of the parsing of the WSDL. Moreover, in my initial implementation, I did not use JAX-WS or any other WebServices technologies like Axis. I simply used an HTTP request to the WebService to get through the issues. This made life a whole lot easier.

The SSL problem reared its ugly head again, but this time I quickly learned 2 things:

1. Exchange creates a default certificate for you when you install it. Either you have to install the certificate as a trusted certificate in the certificate store, or you have to set up a certificate authority on your Windows Server and build signed certificates. I went with the former idea and that resolved the first part of the SSL problem.
2. Exchange is installed to support only NTLM authentication, but NTLM authentication is a pain to work with from non-Microsoft technologies. You can change the settings of the EWS virtual directory on the Microsoft IIS Server that runs on the CAS to support basic and digest authentication. For testing, I set this up as basic, but I now have digest working. I will probably go back and revisit NTLM again later, but it is very hard to find any information to help with solving problems, so I may simply require digest authentication.

Once I got past these issues, creating the calendar item was actually very simple.

OpenEdge through the Java WebService

The next step was to take what I had created and expose it as a Java Web Service. I went with Glassfish V3 as the application server just to test this and in about a half hour I had the Web Service up and running and soapUI was creating appointments through the Java Web Service without any problems.

I then went back to OpenEdge and used its WSDL analyzer to analyze the WSDL. It worked really well. I then built a quick client that called the Java Web Service from OpenEdge and it worked first time.

Next Steps

Although this is now a working prototype, it is exactly that – a prototype. There are several pieces that still have to be proven:

1. I need to subscribe to change updates from Exchange.
2. I need to get to updates for all sales person mailboxes. This means I need to use Exchange Impersonation and that has a number of security risks associated with it.
3. I need to make sure these updates get through to OpenEdge efficiently which means I need a test harness that can generate several thousand appointments a minute and have them update the OpenEdge AppServer without breaking it.

Conclusion

Having said that, I am really excited and very optimistic about how this is turning out. I have learned more about Windows Server, Microsoft Exchange, Active Directory, Exchange Web Services, OpenEdge Web Services, and Java Web Services than I had ever imagined possible.

In the next few weeks I will be providing an update on where I finally ended up with this from an architectural point of view.

Back in January, 2001, I was involved in a project while I was working at Progress Software Corporation that later became known as Progress Dynamics (I’m going to shorten the name to Dynamics for the rest of this article). The whole idea behind Dynamics is that you design your application components and the definition is stored in a repository – a relational database. At run time, the design of the UI is read from the repository and the UI is rendered based on the definition.

In version 2 of the product, we added functionality to allow the user interface to be rendered in a browser using dynamic HTML and JavaScript. The whole idea was to assemble the application once and render it for any platform and ultimately that could have included .NET, Eclipse RCP, and any other imaginable UI.

How does this have anything to do with content management? Well here’s the thing… content management systems essentially store the definition of the content of a web site in a repository – again, a relational database – and render the web pages from the repository. The subtle difference is that the rendering engine for Dynamics is a set of 4GL programs. For a content management system it is a set of programs (various languages including PHP), cascaded style sheets, HTML, XHTML, DHTML and JavaScript.

One of the things that was really cool about Dynamics was the ability to add custom modules to extend Dynamics for your application’s needs. Most of the content management systems have a pluggable architecture that allows you to do the same thing and you can even customize the rendering engine to your own needs.

As with Dynamics, the focus of content management systems is to get the content writer (application developer in the case of Dynamics) to focus on building content rather than worrying about editing CSS, HTML and XHTML.

The joy of the repository in both cases lies in the functionality it provides for searching and organizing the content or application components.

So now you can see why I like the content management system idea. In the same way as I still believe in the core principles of Dynamics – a rendered UI that is portable across multiple technologies – I also believe that content management systems provide the same flexibility for web sites.

I should point out that content management systems are not new. There are many of them out there and there are some very good, expensive commercial ones out there. If you are a large enough enterprise to warrant the expense of the investment in a commercial, 24x7x365 content management system, you should go ahead and look at these. Some of the top contenders in this area are:

• Interwoven
• Percussion
• SDL Tridion
• Vignette

I’m not in that market.

I am a firm believer in the open source initiative and there are a number of very good open source solutions out there. I had also narrowed my focus to something that ran on Linux with Apache, MySQL and PHP (LAMP). There are a number of solutions in this area and given the timeframe that I had committed to, I needed to go on some recommendations.

I settled early on on two alternatives, and I went with them because my web hosting provider supported them both. The alternatives were Joomla and Drupal. Early on it seemed that Joomla was the best solution. It seemed to be really easy to work with and getting content onto the site was snap. I was up and running in no time so I selected Joomla for my personal web-site.

Of course, requirements change and when I left Earthlink in April, I needed to get our company web-site, Intangere, LLC up and running fairly quickly. At this point I started looking for new modules, themes and options for extending Drupal.

That was when I realized that Joomla may not have been the best choice. I then started looking into the extension mechanism for Joomla, and although it can be extended, there are some problems doing it. Drupal has a much larger selection of new modules and themes than Joomla does and it also provides a very extensive, well-documented extension model so that it is relatively easy to build on.

Both of these options require some understanding of how they work. They are not outstandingly intuitive when you first start working with them, but once you get the hang of them, they do the job really well. Drupal just does it a little better.

Intangere, LLC, is a completely Drupal-based site and we are very satisfied with what you can achieve in a fairly short period of time.

It’s very interesting to me that these content management systems are using the same ideas as we had when we built Dynamics back in 2001. There is clearly something to the idea of rendering from a database and it’s interesting to see how each has solved the problems that we had to deal with in Dynamics in different ways.

What is clear is that content management systems are a critical part of what has been called "Web 2.0" – the next generation of the internet – and the thing that makes them really interesting to me is how they provide the one of the core enabling technologies for social media/networking.

The real value in them will come from leveraging the social media component to engage customers in more mutualistic relationships.

A number of people have pitched Drupal to me as a competitor to Joomla. One person wrote:

“When we surveyed content management systems …, Joomla was one of the ones we looked at, but on balance Drupal won out. I recently converted my … site to Drupal and brought up two other sites … on the same code base, but with very different appearance and different features on each. I am very impressed.”

I have spent some time evaluating both of these content management systems and while I have chosen to go with Joomla myself, Drupal could probably have done the job I wanted to get done. My initial post was in no way meant to imply that Drupal was not equaly as capable. There were a couple of things that swayed my vote to Joomla.

1. By the time I looked into Drupal, I already understood how Joomla worked. The argument that Drupal is easier to use therefore did not carry the water for me that it would for others and I felt like Joomla’s flexibility was probably better than Drupal’s provided I was willing to make the investment;
2. I am not scared of getting my hands dirty with PHP, MySQL, CSS and XHTML. That means that if I find an obstacle with Joomla, I have the flexibility to get into it with these technologies if I need to; and
3. I like what Joomla has done to address and caution against security issues within Joomla itself. Security is a very real consideration for any web-site and I think the Joomla team understand this. They may not have all the answers but they definitely understand the issues. You should at least look at the Joomla security site and their security forum before you implement a site using Joomla.

In fact, it is the latter point that made me realize I needed to take down my family site while I work on making sure that I have dealt with any potential security threats. It will be back up in a couple of weeks. 

In a discussion on the issue, someone I was talking to made the statement that they did not want a content management system that required a database because of the administration issues associated with it. To me, that argument does not make any sense because if you go with static HTML instead, you still have the complexity of managing the content and determining what the broken links are. With a good content management system, that process can be easily automated. So although you may not have the expense of the database maintenance, you do have the expense of the static HTML management. Most of the maintenance stuff around my 4 Joomla installations is all automated anyway and I simply check the e-mail that I get on a nightly basis to verify that the backup verification process worked. Moreover, there is so much functionality that you simply get for free that I would hate to have to write manually.

Finally, a gentleman by the name of Jeff Pilant pointed me at a web site that is absolutely invaluable on the security side of things. The site calls itself “Common Weakness Enumeration – A Community-Developed Dictionary of Software Weakness Types” and it enumerates 25 of the most dangerous mistakes that developers make that can create serious security threats in their code. This is a highly recommended read. The information is also available in PDF form and although a lot of this information is available in a number of the security books that are already out there, this is a very valuable and concise reminder.