The Software Gorilla » OpenEdge OpenClient

Exchange Web Services Example – Part 4 – Subscriptions and Notifications

Bruce Gruenbaum — Thu, 08 Jul 2010 06:52:11 +0000

Now that we have succeeded in creating, updating and deleting calendar items and mastered Exchange Impersonation, it's time to turn our attention to having Exchange notify us about what it is doing. Part 4 of this series is going to provide a detailed code walk-through of some code that leverages the Subscription API.

I covered a lot of material about the push subscription mechanism in an earlier post, and if you haven't read it, I highly recommend you go back and revisit it as this article builds on that one. It also builds on part 1, part 2, and part 3 of this series, so you should probably have worked through those, too.

Goals and Background

By now, you know that the reason that I have embarked on this project is to integrate Microsoft Exchange with a CRM application that is in Progress OpenEdge. The first part of the exercise was to figure out how to create calendar items based on data entered into the CRM system, and how to automate that for a large number of sales people without having to log in to each mailbox.

Now that we know how to do that, the next step is handling the situation where the sales person makes a change to a calendar item from Outlook or a mobile device. The item change needs to make it back to the CRM application, and for that, the Exchange WebServices Push Subscription API is going to be essential.

When a change is made to an Exchange Server calendar item, the Push Subscription API can send a message to an HTTP target of your choice. As long as the HTTP target knows how to parse the SOAP message that is sent by Exchange, it can react to the events that it are sent.

In this example, we are going to concentrate on 3 events that occur against objects in the default calendar folder for a mailbox; the Created-, Modified-, and DeletedEvents.

The Subscription Process

By now, the diagram to the left is all too familiar, but it clarifies the process really well, so I am going to walk through it again.

The first step (1.1) is to initiate a call to the Java EWS WebService to register a subscription. The EWS WebService then makes a call to Exchange (1.2) to tell it to call back to the Java HTTP Servlet.

When something happens to an item for which we have registered a subscription, Exchange will call the HTTP Servlet (2.1) with a notification message.

Now the order of what happens next is something that you will want to change for production code, because it should happen in the order illustrated in the diagram, but I have decided to take the simple road with the attached code and process everything in a synchronous way. What follows is the order in which I am doing the processing in the sample code.

After parsing the message notification message that was received (2.1), the code determines if there are any items that need to be processed. A notification message only contains the ItemID of the object that has changed. It does not contain the item itself. This means that we need to call the Exchange Server back to get the item's information (2.3).

Once we have the item's information, we can react to it, and in the example code, this includes calling OpenEdge (2.4) with information about the appointment that has changed. It is really important to note that only notifications that actually affect an item are passed on to OpenEdge. This significantly reduces the processing that OpenEdge has to perform.

Finally, when we are done with the processing, we respond to Exchange with an acknowledgement message (2.2) so that it will continue to send us messages.

If you want more information about the details of this process, refer to my detailed description in the blog posting entitled Exchange Web Services – Subscriptions and Notifications.

It is important to remember that Exchange will poll the Java HTTP Servlet at regular intervals to check that it still wants to receive notifications. The frequency is user-defined as explained in the aforementioned article. These StatusEvents do not require any processing, other than an acknowledgement of receipt back to the Exchange Server (2.2). So the Java HTTP Servlet also performs the function of keeping the subscription alive.

I have also provided a mechanism for unsubscribing from the notifications. This mechanism works by keeping track of all the subscriptions that are currently active, and then returning an Unsubscribe response (2.2) to the Exchange Server after a request to unsubscribe has been received in 1.1.

The Code

Now that you understand how the process works, let's take a look at the code.

First, I need to point out that the download contains 2 zip files. If you are a Java developer with no interest in OpenEdge, you should use the file called EWSAPI_JavaOnly.zip. The difference between this code and the EWSAPI_OpenEdge.zip file is that the latter actually makes AppServer calls to an OpenEdge AppServer, whereas the former simply displays the results of processing the messages in the log file.

The EWSAPI_OpenEdge.zip files can also be installed per the regular instructions, but there are a few additional steps that you have to perform. These are documented in a file called OpenEdge_Setup.pdf that is contained in the downloaded zip file. Essentially, you will need to create a database using a definition that is provided, set up an OpenEdge AppServer that has access to the database, and deploy some OpenEdge ABL code to the AppServer so that it can receive the calls. You will also need to copy your OpenClient runtime libraries to the Glassfish installation so that it can make the OpenClient calls. As I said, all of this is covered in OpenEdge_Setup.pdf.

Like the previous articles, you can download the code and set up your workspace as described in part 2.

What's New?

The EWSAPI project that is included in both the EWSAPI_JavaOnly.zip and the EWSAPI_OpenEdge.zip files contains the following code changes in the com.tsg.ews root package (filenames in italics are new files):

EWSListener.java – This code, which is new, contains the Java code for the Java HTTP Servlet that listens for notifications from Exchange.
EWSImpersonationCalendar.java – This code, which is the code for the Impersonation-based calendar WebService, has been modified to support two new operations – Subscribe and Unsubscribe.

In addition, the com.tsg.ews.calendar package contains the following code changes:

Appointment.java – A minor modification has been made to the retrieve method to support retrieval with a ChangeKey. I also changed the code to use namespaces.

Finally, the com.tsg.ews.service package contains a lot of code changes (again, filenames in italics are new files):

EWSResponse.java – This code, which deserializes responses from the Exchange Web Service, has been altered to support namespaces in the deserialization of the XML. It also now supports a SubscriptionID and a Watermark property.
ExchangeService.java – There are a couple of minor changes to expose some methods outside of the class.

An enumeration and a new class have been added to enhance some of the XML parsing that I have been doing:

EWSNamespaces.java – This is an enumeration of the namespaces used in Exchange WebServices messages.
NamespaceContextImpl.java – This class extends NamespaceContext and is used in namespace resolution of the XPath queries. It makes use of EWSNamespaces to perform this resolution.

The following three classes have been added to support the registration and long-term monitoring of subscriptions:

SubscriptionManager.java – This class manages all subscriptions.
SubcriptionList.java – This class is a collection of subscriptions and is used for serialization and deserialization of all subscriptions.
Subscription.java – This class is used to register a subscription and serialize it and it keeps track of the watermarks for the subscription.

The remaining new classes and enumerations have been added to parse notifications from Exchange Server:

Notification.java – This class is the parent class for each notification. It can be instantiated from an Exchange Web Service SOAP Notification message and it will deserialize the message into its internal objects. The notification, itself, has properties for the SubscriptionId, PreviousWatermark, and a collection of SubscriptionEvents. It also has a flag that indicates whether it is just a status event.
SubscriptionEvent.java – A Notification may include one or more SubscriptionEvent objects. These events will be one of the EventTypes (see the next bullet point) for which the subscription has been registered (CreatedEvent, ModifiedEvent, DeletedEvent, or StatusEvent). After the event has been parsed, it contains properties for EventType, SubscriptionID, Watermark (each event will have a different watermark), TimeStamp, AffectedItem (the item that actually changed), and ParentFolder (the folder that contains the item that changed). If the AffectedItem is a calendar item, the calendar item will be retrieved (if possible) and stored in the Appointment property.
EventType.java – This is an enumeration of the supported event types.
IDHolder.java – When we receive an event, it comes with two elements; the first is either a FolderId or an ItemId, and the second is a ParentFolderId. These nodes contain two attributes; an ID and a ChangeKey. The IDHolder has three properties, two of which map to the ID and the ChangeKey. The third is an IDType, described in the next bullet.
IDType.java – This enumeration contains items for each possible type of ID that can be contained in the IDHolder.

The OpenEdge pieces

If you are using the OpenEdge version of the code (EWSAPI_OpenEdge.zip), the EWSAPI project contains the following additional packages:

com.tsg.common.collections
com.tsg.common.interfaces
com.tsg.dynamicopenclient
com.tsg.dynamicopenclient.data
com.tsg.dynamicopenclient.valueholders
com.tsg.exceptions
com.tsg.ews.openedge

All but the last package are modified copies of the code that accompanies the OpenEdge Dynamic OpenClient Java Example article that I wrote back in November 2009.

The last package contains one class, OpenEdgeCallFactory, that is responsible for making the OpenClient call to the AppServer.

The EWSAPI project for OpenEdge also contains an additional folder, oepieces, that contains the ABL procedure (ewscall.p) that it expects to call and a DF file (test.df) that contains the database definitions for the database table that ewscall.p expects to write to.

I'm not planning to cover the OpenEdge part of this in any detail later in the article. All ewscall.p does is accept a set of parameters and write their values directly to a table in an OpenEdge database. The ewscall.p procedure needs to be deployed to an OpenEdge AppServer. The AppServer needs to be running in STATE-FREE mode against a database that contains the ews_update table that is defined in the test.df file.

Additional Setup

Once you have set up the code and have it running, you need to update the WebService definition for the CalendarService in soapUI. The screenshot at left should help you do this. Right-click on the EWSImpersonationCalendarPortBinding tree node, and select Update Definition from the context menu.

Two additional nodes will be added – Subscribe and Unsubscribe. As you will note from the screenshot, I have created two copies of the RegisterServer request and 8 copies of the Subscribe request.

I have two Exchange servers – One running Exchange 2010 and one running Exchange 2007 SP3. Each Exchange Server has 4 mailboxes that I am using for testing, hence 2 RegisterServer requests and 8 Subscribe requests.

One other thing I would recommend is if you have an MSDN subscription and you can install Outlook 2010, do so. It will make your life a whole lot easier for testing. In Outlook 2010, you can add more than one Exchange mailbox to your list of mailboxes which means that you can work on any of the test mailboxes from just one Windows login. This makes testing much, much easier.

Establishing a Subscription

As I have described, there are two parts to the Subscription API – the Subscription and the Notification. This code walk-through is therefore going to be divided into two main sections covering each of those parts of the API. This section covers setting up the subscription, and the subscription starts at the CalendarService WebService which is in the EWSImpersonationCalendar class in com.tsg.ews.

EWSImpersonationCalendar

This code was included in part 3 of this series, but we have added two additional operations in Lines 76 through 98; Subscribe and Unsubscribe:

@WebMethod(operationName = "Subscribe")
@WebResult(name = "Response")
public EWSResponse subscribe(@WebParam(name = "ServerID") String serverID,
				 @WebParam(name = "UserAccount") String account,
				 @WebParam(name = "CallbackURL") String url,
				 @WebParam(name = "OpenEdgeURL") String oeUrl) {
	Subscription sub = new Subscription(serverID);
	sub.setAccount(account);
	sub.setListenerURL(url);
	sub.setOpenedgeURL(oeUrl);
	EWSResponse resp = sub.subscribe();
	if (!resp.hasError()) {
		SubscriptionManager.getInstance().register(sub);
	}
	return resp;
}

@WebMethod(operationName = "Unsubscribe")
@WebResult(name = "Response")
public EWSResponse unsubscribe(@WebParam(name = "SubscriptionID") String subscriptionID) {
	SubscriptionManager.getInstance().unsubscribe(subscriptionID);
	return new EWSResponse("NoError", 0, "Success");
}

These two operations rely heavily on the concept of the SubscriptionManager which keeps track of all active subscriptions and ensures that they are persisted to a local XML file. This means that when a notification is received later, we have the ability to relate that notification back to the subscription that initiated it, even if we shutdown and restart the Glassfish server. We will definitely talk more about this later once you have a chance to understand the concepts that affect it.

Looking at the Subscribe operation/method in lines 76 through 91, the operation expects to receive a ServerID as an input parameter. This ServerID is the ServerID we retrieved from a previous RegisterServer call. The UserAccount is used for Exchange Impersonation and to specify the mailbox for which the subscription is to be registered. The CallbackURL is the URL of the Java HTTP Servlet that will handle the notifications, and the OpenEdgeURL is the URL that should be used to call the OpenEdge AppServer.

Lines 82 through 85 establish a Subscription object and in line 86, the subscribe method is called. We'll look at the subscribe method in a little more detail in a moment, but for now just accept that it is responsible for actually making the call to the Exchange WebService and obtaining a SubscriptionID. In production code, the Subscription object would be instantiated by the SubscriptionManager, but it makes for more readable example code to do it the way I have done it here.

Lines 87 and 88 check the error status on the EWSResponse object and register the subscription with the SubscriptionManager if the subscription was successful. Whether the subscription was successful or not, the EWSResponse object is returned from the WebService, and if it was successful, it contains the SubscriptionID and Watermark for the subscription.

The Unsubscribe operation in lines 93 through 98 pass the SubscriptionID to the unsubscribe method on the SubscriptionManager.

There is a special case that the Unsubscribe method will support: If an "*" is passed as the SubscriptionID, all existing subscriptions will be canceled.

The Subscription Object

I said we would take a closer look at the subscribe method, and we'll do that in just a moment. First, though, open the Subscription.java class that is in the com.tsg.ews.service package. For the most part, this object is a set of properties, including the ServiceID, ListenerURL, OpenEdgeURL, Account, SubscriptionID and Watermark. The class has JAXB serialization annotations associated with it so that it can be serialized to and from XML.

There is also a canceled flag. The canceled flag is set to false until the cancel method in line 130 is called.

The subscribe method is in lines 97 through 128. The SOAP message that needs to be sent is built between lines 98 through 118:

public EWSResponse subscribe() {
	StringBuffer request = new StringBuffer();
        request.append("\n");
        request.append("    \n");
        request.append("\n");
        request.append("        \n");
        request.append("            ");
        request.append("        \n");
        request.append("        \n");
        request.append("            CreatedEvent\n");
        request.append("            ModifiedEvent\n");
        request.append("            DeletedEvent\n");
        request.append("        \n");
        request.append("        3\n");
        request.append("        ");
        request.append(this.getListenerURL());
        request.append("\n");
        request.append("    \n");
        request.append("\n");
        SOAPRequest message = new SOAPRequest(request.toString(), this.getAccount());

The Subscription message has been discussed in some detail in my previous post on Subscriptions and Notifications, and this message is largely the same message as that one, with the exception that the user can specify the URL, and the instantiation of the SOAPRequest object in line 118 (the last line) results in the use of Exchange Impersonation to make the request.

The DistinguishedFolderId node that is added in line 105 will result in notifications being generated based on any changes to the folder(s) that is(are) listed – in this case, just the calendar folder. The following folders are supported by name:

calendar
contacts
deleteditems
drafts
inbox
journal
notes
outbox
sentitems
tasks
msgfolderroot
root
junkemail
searchfolders
voicemail

Any folder can be monitored, but if it is not one of these folders, you need to supply the FolderID for the folder.

The EventTypes node in lines 107 through 111 defines the list of events for which notifications should be received. This list could include any of the following values:

CopiedEvent
CreatedEvent
DeletedEvent
ModifiedEvent
MovedEvent
NewMailEvent
FreeBusyChangedEvent (Exchange 2010 only)

The StatusFrequency node is described in a lot more detail in my previous post. In this case, I am asking for Exchange to send a status event to the Java HTTP Servlet every 3 minutes. This gives me the opportunity to cancel the subscription every 3 minutes. I have mixed feelings about status frequencies in terms of how long they should be, and I'll talk more about this later in this article.

Once the message is built, we call makeRequest on the ExchangeService object with the message in lines 119 through 128:

        EWSResponse response;
        try {
		response = getService().makeRequest(message);
		subscriptionID = response.getSubscriptionId();
		watermark = response.getWatermark();
	} catch (Exception e) {
		response = new EWSResponse(e);
	}
	return response;
}

If the call is successful, the EWSResponse object will contain a unique SubscriptionID and a Watermark. It is important to understand that the SubscriptionID is unique, even if you have already registered a subscription for this mailbox. In other words, it is possible to have more than one subscription active for a mailbox.

The SubscriptionManager Object

The SubscriptionManager object is responsible for tracking all active subscriptions. In essence, it is a persistence mechanism for maintaining the state of a subscription over a long period of time. The SubscriptionManager takes care of persisting the subscription details to an XML file and providing a central point through which all changes to a subscription can take place. Any time any object needs information about a subscription, it makes a call to the SubscriptionManager to obtain information about the subscription.

Any changes made to a subscription are persisted to the XML file, so that if the Glassfish server is restarted, and the subscription is not lost, the SubscriptionManager will be able to maintain the context of any subscriptions that are currently active.

Why do we care about this context anyway? For a number of reasons:

We associate the OpenEdge URL with a subscription and there is no way to include this information with the information that we submit to Exchange Server so there is no way to know what we need to do when we receive a notification if we don't persist the context of the subscription;
To be able to specifically cancel a subscription, the SubscriptionManager needs to know about it. Remember that the only ways to cancel a subscription are by responding to a notification with an unsubscribe response, or by not responding and waiting for it to time out. Ideally, we need the ability to do this gracefully, and that means responding to a notification with the unsubscribe response. It may be several minutes between the time that an unsubscribe request is received by the Java EWS WebService and the time a notification is received by the Java HTTP Servlet, so all the SubscriptionManager can do is note that the subscription is canceled. When the next notification comes in, the Java HTTP Servlet can then act on the cancellation;
In a production environment where there may be numerous mailboxes that are all being monitored concurrently, it would be far more efficient to receive all notifications for one mailbox as a set. The problem is that when a subscription is registered, there may be no way of knowing what other subscriptions will be registered for the same mailbox. Although the implementation that I have included does not do this, the SubscriptionManager is the ideal place to maintain a link between subscriptions for the same mailbox so that the load on the Exchange Server can be minimized and delegated to a separate middleware service; and
If the Glassfish server goes down for any reason, context needs to be maintained so that when it is restarted, it can re-establish the subscriptions as they were running before it was terminated. Of course, in a production environment with load balancing in place, it would be unlikely that the Glassfish server farm would not be available, but the notion of more than one Glassfish server immediately means that the subscription context needs to be persisted centrally.

So although the SubscriptionManager that is provided with this example is somewhat primitive, it performs a key role in maintaining continuity and performance for subscriptions.

The SubscriptionManager code is not particularly interesting so I am not going to spend any time walking through it. It definitely has holes in it, but it works for the examples that we are using here.

Establishing a Subscription

Now that you know how the code works for establishing a subscription, let's go ahead and register a subscription for one of the mailboxes. Assuming you already have the code installed and running in Eclipse and the Glassfish server running, open soapUI and open a request under the Subscribe operation for the CalendarService:

The ServerID element should contain a UUID for a server that was registered using the RegisterServer operation. The UserAccount should contain the e-mail address of the mailbox that you want to monitor.

The CallbackURL is the first new element you need to complete. This is the URL that the Exchange Server should use for all notifications. This URL will be in the form http://:8080/EWSAPI/EWSListener where is the machine that is running your Glassfish server (your local machine). Note that you cannot use localhost as the server name. Calls to this URL are going to come from your Exchange Server so it needs a DNS-registered name.

The OpenEdgeURL is only of interest to you if you are an OpenEdge developer. If this element contains anything but a string that begins with AppServer:// it will be ignored. If you are using the default asbroker1 AppService, the URL that I have in the screenshot (AppServer://localhost:5162/asbroker1) should work for you, assuming your AppServer is running on the same machine as your Glassfish server.

Once you have filled out the request parameters, submitting the request will get you a response similar to the one in the following screenshot:

The Response node contains 3 attributes. The ErrorState attribute should have a value of "NoError" and should be followed by a SubscriptionID attribute and a Watermark attribute, both with strings of characters that don't mean anything. The SubscriptionID uniquely identifies the subscription and can be used to unsubscribe through the Unsubscribe operation.

If you switch your focus to the Console view in Eclipse for the Glassfish Server Log, you will see messages start popping up occasionally in the view like the one below:

INFO: Notification:
=============
	Account: froggf@intangere.internal.intangere.com
	Subscription Id: LQBpbnRhbmdlcmVleHMuaW50YW5nZXJlLmludGVybmFsLmludGFuZ2VyZS5jb20QAAAAdyIIAinbTUGf+zpPjEyTaw==
	Prev Watermark: AQAAANnz8X+xbA5HtnQ8PXPRJfHGLgAAAAAAAAE=
	More Events: false
	Status: true
	Event: StatusEvent	Watermark: AQAAANnz8X+xbA5HtnQ8PXPRJfHGLgAAAAAAAAE=

INFO: 
INFO: Total execution time: 205

These messages will pop up every 3 minutes and indicate that the Glassfish server is receiving messages from the Exchange Server.

If you switch to Outlook and create a calendar item in Outlook, you will see a message something like the one below about 20 seconds after you create it:

INFO: Notification:
=============
	Account: froggf@intangere.internal.intangere.com
	Subscription Id: LQBpbnRhbmdlcmVleHMuaW50YW5nZXJlLmludGVybmFsLmludGFuZ2VyZS5jb20QAAAAdyIIAinbTUGf+zpPjEyTaw==
	Prev Watermark: AQAAANnz8X+xbA5HtnQ8PXPRJfHGLgAAAAAAAAE=
	More Events: false
	Status: false
	Event: ModifiedEvent	Watermark: AQAAANnz8X+xbA5HtnQ8PXPRJfHJLgAAAAAAAAE=
			TimeStamp: Wednesday, July 7, 2010 6:39:42 PM PDT
		Affected Item-- Type: FolderId	ID: ...	ChangeKey: AgAAAA==
	Event: ModifiedEvent	Watermark: AQAAANnz8X+xbA5HtnQ8PXPRJfHKLgAAAAAAAAE=
			TimeStamp: Wednesday, July 7, 2010 6:39:42 PM PDT
		Affected Item-- Type: FolderId	ID: ...	ChangeKey: AgAAAA==
	Event: ModifiedEvent	Watermark: AQAAANnz8X+xbA5HtnQ8PXPRJfHLLgAAAAAAAAE=
			TimeStamp: Wednesday, July 7, 2010 6:39:42 PM PDT
		Affected Item-- Type: FolderId	ID: ...	ChangeKey: AgAAAA==
	Event: CreatedEvent	Watermark: AQAAANnz8X+xbA5HtnQ8PXPRJfHRLgAAAAAAAAE=
			TimeStamp: Wednesday, July 7, 2010 6:39:42 PM PDT
		Affected Item-- Type: ItemId	ID: ...	ChangeKey: DwAAAA==
		Appointment-- Subject: New Appointment for Software Gorilla test
			Location: Who cares where! Just make sure it works!!
			Start: Wednesday, July 7, 2010 7:00:00 PM PDT	End: Wednesday, July 7, 2010 8:00:00 PM PDT

INFO: 
INFO: Total execution time: 75

I have edited this slightly so it fits on the screen. This notification informs you that the Exchange Server has called the Java HTTP Servlet which has processed the notification.

If you supplied an OpenEdge URL for the AppServer, and you connect to the database that contains the ews_update table from OpenEdge and execute the following code, you will find entries in the database for each appointment that you created:

for each ews_update:
    display skip(1)
        ews_update.cEmailAddress skip
        ews_update.cSubject skip
        ews_update.cLocation skip
        ews_update.dTimeStamp skip
        ews_update.dStartTime ews_update.dEndTime skip(2)
      with 1 down side-labels width 100.
end.

There are a couple of important things to note about the data in the OpenEdge database. If you are not living in the UTC timezone, you will see a difference between the time that OpenEdge reports and the time that notification reported. That's to be expected. Time in the database is reported as UTC. In my case, I am in the PDT (Pacific Daylight Time) timezone.

Second, it looks like the subject and location have been truncated. Again, no surprise. I set the formats in the database fields to X(50) for the subject and location, although the database will store more than that for them.

Looking back to the notification that we received from Exchange, you'll note that there is a Watermark for each event, and that each Watermark is different (sometimes by only one character). The Watermark provides a way of going back and verifying that you have received all events from Exchange, but it is important to realize that subscribing with an old Watermark can have significant performance impacts on Exchange.

You'll also notice that there is a Total Execution Time for each notification. This is the time taken (in milliseconds) from the time the notification was received, until the time the response was flushed out to the HTTP stream. In other words, it is the total time taken to process the request. What is interesting about this is that it takes as much as 4 times longer to process a StatusEvent than it does to process a real notification. I have determined that this has to do with the time it takes to read the original notification message off the input stream, and I have not yet figured out how to shorten that time.

Now that you have seen the notification mechanism working, let's look at what is going on under the covers.

Receiving Notifications – EWSListener

As noted, notifications are received on the http://:8080/EWSAPI/EWSListener URL. To achieve this, the EWSListener.java class in com.tsg.ews listens on the URL for incoming requests, so go ahead an open EWSListener.java in Eclipse.

All of the work associated with processing a request occurs in the doPost method that runs between lines 52 and line 170. I have included a lot of comments in the code that is in the zip file, so I am going to extract sections of it here and exclude the comments that are in the source. Line numbers, though, correspond with the source in the zip file.

Receiving the Notification (2.1 in the diagram above)

Lines 52 through 63 deal with receiving the notification:

protected void doPost(HttpServletRequest request,
		HttpServletResponse response) throws ServletException, IOException {
	boolean safeDoc = false;
	boolean canceled = false;

	long startTime = GregorianCalendar.getInstance().getTimeInMillis();
	long endTime;

	try {

		String result = ExchangeService.getResult(request.getInputStream());
		Document doc = loadXML(result);
		safeDoc = true;

		Notification nfcn = null;
		try {
			nfcn = new Notification(doc);

The notification XML document is received from Exchange in the request object. The response object will contain our response back to Exchange later.

The safeDoc flag is used to indicate that we have successfully parsed the XML document and that it is indeed a document that we expect. The canceled flag indicates that the response to Exchange should be an unsubscribe message. The startTime and endTime fields are used to calculate the total elapsed time of the call.

In line 63, we make a call to getResult (now a static method) on the ExchangeService class. This method simply reads all the data off an input stream.

Now that we have a character string, it's time to do something with it, and in line 66, we make a call to the internal loadXML method which parses the XML string into a DOM document.

In line 74, we instantiate a Notification object with the XML document and the Notification object takes care of parsing the the DOM into appropriate objects.

Parsing the XML

I'm not going to spend any time walking through the code that parses the XML document, but the following XML is from an actual notification for the creation of a new appointment (I've truncated IDs for brevity):



    
        
            
                
                    NoError
                        
                            LwBncnzAg=
                            AQAAAAE=
                            false
                            
                                AQAAAAAE=
                                2010-07-05T18:06:29Z
                                
                                
                            
                            
                                AQAAAAE=
                                2010-07-05T18:06:29Z

The Notification element contains a SubscriptionId, PreviousWatermark and MoreEvents element. When the code parses this XML document, it creates a Notification object to contain this information.

The Notification element also contains an Event element for each event for which you are receiving a notification. In this case, the first Event element is a ModifiedEvent and it applies to a Folder (in this case, the calendar folder that has been modified by the addition of a new item). The second event, the CreatedEvent element, contains an ItemId element that identifies the calendar item that was created.

The Notification is an iterable collection of SubscriptionEvent objects, so when the code parses these elements, it creates and stores a SubscriptionEvent object to hold each of these Event elements. Each Event node contains a Watermark that is applicable for the event that took place and a TimeStamp – the time that the event took place. These values are recorded against the SubscriptionEvent object.

An Event element will contain either a FolderId element or an ItemId element which is the object that was affected by the event. This element is parsed and an IDHolder object is created to contain it and it is stored as the AffectedItem property of a SubscriptionEvent object.

There is also a ParentFolderId which is the ID of the folder that contains the object that is affected by the event. When this element is parsed, its data is also stored in an IDHolder object in the ParentFolder property of the SubscriptionEvent object.

In the case of the above notification, one Notification object was created and it contained two SubscriptionEvent objects – one for the ModifiedEvent and one for the CreatedEvent.

Marrying the Notification and the Subscription

Now that we have parsed the notification, we'll go back to the code in line 77 through 86 and deal with marrying the Notification with a Subscription object:

	Subscription sub = SubscriptionManager.getInstance().getSubscription(nfcn.getSubscriptionId());
	if (sub != null) {
		nfcn.setAccount(sub.getAccount());

		canceled = sub.isCanceled();
		if (canceled)
			SubscriptionManager.getInstance().clearSubscription(sub);

		if (!nfcn.isStatus()) {
			getAppointments(sub.getService(),nfcn);
		}

In line 77 we ask the SubscriptionManager for the Subscription object associated with the SubscriptionId in the Notification object. The next set of code only executes if we found a Subscription object.

In line 80, we set the account (e-mail address) for the notification to the account of the subscription and in line 84, we check to see if the subscription has been canceled by a call to the Unsubscribe operation. If it has, we set the canceled flag and remove the subscription from the SubscriptionManager because this is the last notification we will receive for this subscription.

As I mentioned earlier, Exchange will send notifications on a regular basis (as defined by StatusFrequency). These notifications do not contain any data; they are simply status messages (think of them as pings). When we parse the SOAP message into the Notification object, if the notification is a StatusEvent, we set a flag to indicate that this is only a status message. In line 91 we check to see if this is a status message, and if not, we make a call to getAppointments which is responsible for retrieving appointments from the Exchange Server.

getAppointments (2.3 in the diagram above)

The getAppointments method is in lines 176 through 209:

private void getAppointments(ExchangeService service, Notification nfcn) {

	for (SubscriptionEvent se: nfcn) {

		if (se.getEventType() != EventType.STATUS
				&& se.getEventType() != EventType.DELETED
				&& se.getAffectedItem() != null
				&& se.getAffectedItem().getIdType() == IDType.ITEM) {

			Appointment appt = new Appointment(service);
			appt.setEmailAddress(nfcn.getAccount());

			EWSResponse resp = appt.retrieve(se.getAffectedItem().getId(),
                                                  se.getAffectedItem().getChangeKey());
			if (!resp.hasError()) {
				se.setAppointment(appt);
			}
			else {
				if (resp.getResponseCode().equals("ErrorItemNotFound")) {
					se.setEventType(EventType.DELETED);
				}
				else {
					System.out.println(...);
				}
			}
		}
	}
}

As you will recall, the notification that we receive from Exchange only contains the ID of the item that has changed. It does not provide any information about the item itself. In getAppointments we take care of retrieving those items.

Notification objects are iterable collections of SubscriptionEvents so we loop through all the SubscriptionEvents (line 179) in the Notification collection, ignoring anything that is not a Created- or ModifiedEvent for an ItemId (lines 183 to 186).

We then instantiate an Appointment object (line 189), set the account for it (line 190), and attempt to retrieve it from the Exchange Server (line 193).

If we're successful retrieving the appointment, we set the Appointment property of the SubscriptionEvent from the appointment we retrieved (line196).

If we failed with an ErrorItemNotFound message from Exchange, it means that the item has been deleted so we cannot find any information about it. The event type therefore needs to be changed to a DeletedEvent (lines 200 through 202). This may seem strange, but remember that Exchange will move a deleted item to the Deleted Items folder which means we will receive a ModifiedEvent, rather than a DeletedEvent.

Calling OpenEdge (2.4 in the diagram above)

Returning to the code in doPost, lines 96 through 103 deal with the call to OpenEdge:

	System.out.println(nfcn.print());

	// **OESPECIFIC
	// The following code is inactive in the Java zip file version of the code.
	// If the subscription has an OpenEdge AppServer URL in it, let's call OpenEdge.
	if (sub.getOpenedgeURL().startsWith("AppServer://")) {
		OpenEdgeCallFactory.callOpenEdge(sub, nfcn);
	}

Line 96 just displays the notification information in human readable form – it is responsible for the Notification messages in the server log.

Line 101 through 103 checks if the OpenEdge URL for the subscription starts with the string "AppServer://" and, if so, calls the callOpenEdge method which is in lines 68 through 99 of the OpenEdgeCallFactory class:

public static void callOpenEdge(Subscription sub, Notification nfcn) {
	checkInitialized();
	for (SubscriptionEvent se: nfcn) {
		if (se.getAffectedItem() != null && se.getAffectedItem().getIdType() == IDType.ITEM) {
			Call call = new Call(OECALL);
			call.setParameterValue(PAR_EMAIL, sub.getAccount());
			call.setParameterValue(PAR_SUBSCRIPTION_ID, nfcn.getSubscriptionId());
			call.setParameterValue(PAR_EVENT_TYPE, se.getEventType().toString());
			call.setParameterValue(PAR_TIME_STAMP, se.getTimeStamp());
			call.setParameterValue(PAR_WATERMARK, se.getWatermark());
			call.setParameterValue(PAR_ITEM_ID, se.getAffectedItem().getId());
			call.setParameterValue(PAR_CHANGE_KEY, se.getAffectedItem().getChangeKey());
			Appointment appt = se.getAppointment();
			if (appt != null) {
				call.setParameterValue(PAR_SUBJECT, appt.getDescription());
				call.setParameterValue(PAR_LOCATION, appt.getLocation());
				call.setParameterValue(PAR_NOTES, appt.getNotes());
				call.setParameterValue(PAR_START_TIME, appt.getStartDate());
				call.setParameterValue(PAR_END_TIME, appt.getEndDate());
				call.setParameterValue(PAR_REMINDER_MINUTES, appt.getReminderMinutes());
				call.setParameterValue(PAR_REMINDER_SET, appt.isReminderSet());
			}
			call.executeCall(sub.getOpenedgeURL());
			if (call.isReturnError()) {
				System.out.println(...);
			}
			else if (call.getFailure() != null) {
				System.out.println(...);
			}
		}
	}
}

This method simply instantiates a Call object (line 72), sets the parameters for the call to ewscall.p (lines 73 through 89), and executes the call using the OpenEdge URL (line 90). A successful call to ewscall.p will result in the data being written to the OpenEdge database. If you read my article on the OpenEdge Dynamic OpenClient Java Example, this code will be reasonably familiar.

At this point, all that is left to do is update the subscription and respond to the Exchange Server.

Updating the Subscription

From the information in David Sterling, et al's book, Inside Microsoft Exchange Server 2007 Web Services, I had understood that the Watermark was associated with a subscription, but in working with this a little more, I have come to the conclusion that the Watermark is associated with the Exchange Server as a whole. I had been hoping to use the Watermark to guarantee that I knew that I had received all events from the Exchange Server, but it does not seem to work the way I had understood. I'm still piecing this together at the time of writing, so the code that is included with the example is predicated on my understanding from Sterling, et al.

Lines 107 through 127 deal with updating the subscription:

			if (!sub.getWatermark().equals(nfcn.getPreviousWatermark())) {
				System.out.println(...);
			}

			String lastWm = null;
			for (SubscriptionEvent se : nfcn) {
				lastWm = se.getWatermark();
			}

			if (lastWm != null) {
				sub.setWatermark(lastWm);
			}
		}
		else {
			canceled = true;
			System.out.println(...);
		}
	}

In line 107 through 109, we check to see if the subscription's current watermark matches the previous watermark of the notification we received. If it doesn't (and if my understanding of Sterling, et al is correct, it should), we print a warning message to the log.

In lines 112 through 116, we loop through all the SubscriptionEvent objects and store the last watermark and line 117 through 119, we set the Watermark on the Subscription to the Watermark of the last SubscriptionEvent.

In lines 121 through 125 we deal with the situation where we were unable to match the notification with a subscription – we set the canceled flag to true so that the subscription will terminate.

Responding to Exchange (2.2 in the diagram above)

In lines 138 through 170 we deal with responding to Exchange:

	if (!safeDoc) {
		response.sendError(HttpServletResponse.SC_BAD_REQUEST);
	}
	else {
		String str;

		if (canceled) {
			str = getResponseXML(UNSUBSCRIBE);
		}
		else {
			str = getResponseXML(OK);
		}

		response.setCharacterEncoding("UTF-8");
		response.setStatus(HttpServletResponse.SC_OK);
		response.setContentType("text/xml; charset=UTF-8");
		response.setContentLength(str.length());

		PrintWriter w = response.getWriter();
		w.print(str);
		w.flush();
	}
	response.flushBuffer();
	endTime = GregorianCalendar.getInstance().getTimeInMillis();
	System.out.println(String.format("Total execution time: %1$s", (Long) (endTime - startTime)));
}

In lines 139 through 141, we handle the situation where we received a bad message from Exchange – we return an HTTP 400 error.

In lines 143 through 151 we check to see if the canceled flag is set. If it is, we build an Unsubscribe response, otherwise we just build an OK acknowledgement response. Lines 154 through 157 set some properties of the response object and lines 160 through 167 write the response to the output stream and flush the buffers.

Finally, in lines 168 and 169 we determine the amount of time that has elapsed since the notification was received and we display this.

Summary

That's it. Now you have seen how to handle the Exchange Web Services Push Subscription API in Java. Although this may seem like a convoluted piece of code, I have found it to be extraordinarily stable. I have experimented with taking down the Glassfish server and bringing it back up and the notifications keep on coming.

I have left the server running for days with no updates happening on the Exchange Server with no problem at all. I have also exposed it to some pretty serious stress and it has performed with no problems at all on fairly small hardware and software configurations.

Most of the time, notifications, including the call back to Exchange Server and OpenEdge, are being processed in around 100ms. StatusEvents take just over 200ms.

The one thing I am still having trouble with is trying to decide on an appropriate StatusFrequency. I would like to set the StatusFrequency to 10 minutes. That way, it will take 30 minutes for a subscription to expire. The problem is that clean cancellations rely on regular status events, so anything longer than 3 minutes can seem like an eternity. One thing I have definitely concluded is that the StatusFrequency needs to be configurable.

I hope this article has been useful. The next article in this series is going to deal with determining availability, but I am going to be engaged in some pretty serious development work for the next few weeks so it is likely that the next article in this series will only see the light of day toward the end of the Summer.

Let me have your feedback…

Exchange Web Services – Subscriptions and Notifications

Bruce Gruenbaum — Tue, 27 Apr 2010 01:15:00 +0000

It's been a really busy week since I posted my first post on Exchange Web Services (EWS). I have learned a lot in that short period of time that I want to share with you. Whether you are an OpenEdge, Java or .NET developer, I think this post is going to have some information for all of you.

In my first post, I told you about the background story – I need to enable an OpenEdge CRM application to create, modify and delete calendar and task items in Microsoft Exchange. I also need Exchange to let me know any time a calendar or task item is changed so that I can update the OpenEdge database accordingly. Simple use cases.

When I left off last week, my next step was to get Exchange subscriptions working, and, boy, what a trip that has been.

Types of Subscriptions

There are two types of subscription models available with Exchange:

Pull Subscriptions – You set up a subscription with Exchange and it gives you an indicator (called a Watermark) that keeps track of what events you have already received. You pole the server at an interval and it gives you a response with all events that have happened since that last Watermark.
Push Subscriptions – You set up a subscription with Exchange and tell it to call you back on another WebService every time something happens. The WebService has to acknowledge receipt of the message in order to keep the subscription alive. The message you get contains a list of items that have been affected since the last time a message was acknowledged.

I'm not interested in polling the server so I have chosen to go with a Push Subscription. Much of the information that follows applies to both subscriptions, but the Push Subscription differs in that you do not poll the server for the changes.

Setting up a Push Subscription

To set up a Push Subscription, you send a SOAP message to EWS, authenticating against the Active Directory as you would with any other message to EWS. The message contains a list of folders that you want to know about, the list of event types that you want to be notified of, the URL that you should be called back on, and a StatusFrequency node that tells it how often to poll the URL – more about this later.

You can subscribe to virtually any update on virtually any folder in a mailbox, including the calendar, tasks, contacts, and mail folders.

The diagram to the left shows the architecture that I finally ended up with. It is a little more complicated than it would be if OpenEdge were not in the picture, but it serves to explain the process, nonetheless. The numbers in parenthesis below refer to the numbers in the diagram.

OpenEdge initiates the subscription by sending a message (1.1) to a Java WebService that is deployed on a Glassfish Application Server. The WebService forwards that subscription request (1.2) on to the IIS server that is running on the Windows 2008 Server that hosts the Microsoft Exchange Client Access Server (CAS). IIS passes the message onto the EWS API which registers the subscription with the CAS. CAS responds with a message that contains the Subscription ID and the Watermark which is tied to the last event that was sent.

As I mentioned, the message that is sent to the EWS API (1.2) contains the URL for the service that is to receive all event notifications. It also contains the StatusFrequency node.

I set up a servlet on my Glassfish Application Server to receive and respond to the notifications. The following is a sample subscription request message:


    
        
            
                
                    
                        
                    
                    
                         CreatedEvent
                         ModifiedEvent
                         DeletedEvent
                    
                    3
                    http://server.example.com/ewsCalTest/NotificationService

The StatusFrequency Node

The StatusFrequency Node tells EWS to send a message every x minutes to verify that the subscription is still active. This serves three purposes:

It allows Exchange to figure out if the server that is listening is still around and cancel any dead subscriptions if necessary;
It allows the server that is being called to determine that it is still getting messages from the Exchange Server; and
It allows the server that is being called to unsubscribe from any future messages.

Exchange needs to carefully manage resources, so canceling any unnecessary subscriptions is an important way to do that. To maintain a subscription, the server that is receiving the notifications has to respond with a SOAP message that includes a SubscriptionStatus node with the value "OK".

The expected response is as follows:

OK

If a subscriber does not respond appropriately, EWS goes into retry mode. In other words, if EWS does not get a SOAP message formatted exactly the way it expects, it assumes the server has not received the message. It waits for 30 seconds and tries again. If it still receives no response, it waits for a minute and tries again. EWS will continue retrying, doubling the wait time, until the wait time exceeds the StatusFrequency. So in my case, when I first set this up, I set the StatusFrequency to 3 minutes. It tried to send the notification and waited 30, then 60, then 120 seconds and canceled the subscription because, if it had gone to 240 seconds, it would have exceeded the 3 minutes set as the StatusFrequency.

Less obvious, though, is the reason behind purposes 2 and 3. Remember that Exchange is only going to send messages when something changes. Well, some users have very quiet mail boxes. They may be out of the office, or it may simply be after hours where nothing is going on. If Exchange does not let the other server know that it is still sending messages, how is that server to know that the subscription is still valid?

The other thing that is important is that there is no way to send a message to Exchange to tell it to cancel a Push Subscription. The only way to cancel the subscription is by responding to a status message with the SubscriptionStatus set to "Unsubscribe".

Receiving Notifications

Now that you understand how to subscribe, the next step is to start receiving messages from EWS. We'll pick up where we left off in the diagram above.

EWS starts off by sending a notification (2.1) to the server. The notification could be one of two things:

A notification of events that have taken place on the server; or
A status message.

If this is a status message, all you need to do is respond with the OK response (2.2) I discussed above. In fact, you need to respond with the OK message if you want to carry on receiving notifications no matter what, but responding with an OK has an additional effect if you receive a notification of events.

First let's understand a notification of events. An event notification message contains a list of events that have taken place against the items that you are monitoring. It does not contain the details of what changed – just the fact that something did. If you want to know what changed, you have to call the GetItem method of the EWS to get the item and look at its changes. That's what we do in step 2.3.

Each notification includes a PreviousWatermark for the whole message and a Watermark for each event. By keeping track of the watermarks, it is possible to restart the subscription from the last event that was processed. When you acknowledge the event notification with an OK, Exchange moves the subscription to the next watermark automatically.

Once I have processed the event notification and received the event, I am making a Dynamic OpenClient call (2.4) to the OpenEdge AppServer to notify the AppServer of the change. The ABL code on the AppServer can then do what it needs to do to respond to the event.

Performance

Once I got this working, I decided to test it out to see what the performance looked like. Now bear in mind, all of the machines shown in the diagram are different virtual machines on one Dell Poweredge T710 running VMWare 4.0 ESXi. The T710 has 2×4 core processors and 16GB of memory. It has 4 network cards connected to a 1GB switch. The Windows Server is on a separate network card to the other three machines, which are all Centos 64-bit boxes.

The round trip for establishing the subscription takes an average of 82ms.

I then ran a test harness that generates 100,000 calendar items across 80 mailboxes. It took a little over 2 hours for the test harness to complete. I tracked performance for each of the following pieces:

The OpenEdge client takes an average of 100 milliseconds to make the call through the Java WebService to EWS to create the appointment.
From the time I receive the acknowledgement for the appointment creation to the time that the notification has been received (2.1) and acknowledged (2.2) and the details of the notification have been received (2.3) takes an average of 150 milliseconds.
The call across the Dynamic OpenClient to the AppServer, which is a call to a procedure that has no temp-tables or XML documents, averages 80 milliseconds.

All told, this means that from the time the request for the appointment is initiated to the time the notification of its update is received by the AppServer takes an average of 330 milliseconds – less than half a second.

Now clearly, my environment has very little traffic and I have a lot of horsepower backing it, but it if this is properly tuned and the additional processing that is to be added is not excessive, a one second response time is well within the realms of possibility.

Problems and Challenges

In getting this to work, I ran into a couple of nasty obstacles.

First, Microsoft's API is really not a good example of a WebServices API. Neither Java nor OpenEdge will parse the WSDLs without manual editing and even if you manage to generate the code you need from the WSDL, it really doesn't work very well. I have been forced to manually code the interaction with the EWS API and that is a lot of work. Having said that, it performs very well and I get to write the XML exactly the way it needs to be for EWS to accept it.

That brings me to the second thing about this API. Microsoft's error reporting is shockingly bad for commercial software. Compared to the kind of error reporting I get from Tomcat, Glassfish, ActiveMQ, and several other Open Source solutions, it's incredible that commercial software can be this badly engineered. It is so hard to figure out what is wrong when something goes bad that you will spend hours on things like malformed or unrecognized XML. EWS simply returns a 500 http error code (Internal Server Error) with absolutely no indication of what caused it, and there is nothing in any of the logs to help you.

Update: Since I wrote this article, I have found (and Kris C. has also pointed this out in the comments) that making a call to the getErrorStream() method on the connection object will provide more detailed information on what failed, and you get an entire SOAP message back that describes details of the 500 error. I therefore eat my words about error reporting.

I still think that there is room for improvement because it is very hard to diagnose exactly which node is out of order if you submit a request where the XML nodes are not in the exact order specified by the EWS API, but it's nowhere near as bad as what my original post probably led you to believe.

There is more information in Part 3 of the series and there is even sample code that demonstrates exception retrieval.

The third thing that really cost me a lot of time is that you cannot call a regular Java or OpenEdge WebService from EWS because it won't look at the WSDL. I was therefore forced to manually code the Java servlet that accepts an http post.

These problems effectively sidelined OpenEdge as a direct endpoint for EWS notifications. The architecture that I am building will definitely rely on all communication with EWS going via Java services.

Where to next?

Now that I have the prototype code working, my next step is to fix up some of the security issues I need to resolve. NTLMv2 authentication remains a real problem for Java in communicating with EWS, and I am looking for a solution around that. I also need to deal with some of the certificate related issues that I have.

I have also done some work on the Exchange Impersonation stuff, so my next blog post will probably talk about this.

Finally, I'm planning a walkthrough of some example code that actually demonstrates the functionality, like I did with the Dynamic OpenClient code late last year.

Exchange Web Services – Starting out

Bruce Gruenbaum — Tue, 20 Apr 2010 12:28:25 +0000

Many people make use of different mechanisms for creating e-mail messages. Most of them use some kind of SMTP client to do so. For the more adventurous, there is MAPI that allows you access to a Microsoft Mail based client to do a few things over and above just plain mail. These are things that people do on a daily basis.

A couple of months back, a gentleman who has now become a friend and business partner, Jim Ford of Ford Audio-Visual, came to me and asked me if there was any way to get at all the calendar items in his sales organization's calendars with the intention of integrating it with his Progress OpenEdge CRM system. Jim is using Exchange 2007 for his e-mail and calendaring solutions.

I was aware that Microsoft had released a new API for Exchange in Exchange 2007 called Exchange Web Services (EWS), and so I said that I needed to do a little research on the API, but I was pretty sure that it was possible. Sure enough, MSDN has some documentation of the API and Microsoft is touting it as the replacement for all APIs that communicate with Exchange. Web Services – how hard can it be?

The Production Environment

In the production environment, an OpenEdge client running on a Unix server needs to initiate a WebService call to Exchange to create, update and delete appointments for sales people. These appointments will be created in the user interface for the CRM application.

Sales people then need to be able to change and delete these appointments from their Outlook clients or their Blackberry/iPhones. Any changes to the appointments need to be recorded as history updates in the CRM system, so Exchange needs to notify the OpenEdge CRM system of these changes.

So the updates to appointments need to happen from both directions and the communication is going to happen cross-platform (Unix to Windows and vice-versa).

Initial Idea

My initial plan was to create a prototype that did the two-way communication directly from OpenEdge to EWS. I just wanted to prove that it was possible. The first step was to build an OpenEdge WebServices Client that created a calendar item in the Exchange store. Unfortunately, for various reasons, this is much harder than I thought it would be.

First, the EWS WSDL is not a standard WSDL. EWS is installed as part of the Client Access Server (CAS) component of Exchange. In production, you can have multiple Mailbox Servers and the CAS has to discover which mailbox server services a client. As a result, the URL that is used to actually call EWS is different depending on which Mailbox Server hosts the mailbox. The WSDL therefore does not contain a "service" node and the OpenEdge WSDL parser cannot parse the WSDL. I tried working around that by creating a dummy WSDL with a service node inside it.

The second problem was that I ran into all kinds of problems with the complex type structure that exist in the WSDL. The types and messages live in a separate XSD file and the WSDL parser did not like some of the message definitions as they included optional types.

The last straw was that the connection to EWS is an SSL connection that, by default, wants to use NTLM authentication and I battled for hours to get this to work. I would probably have figured the problem out earlier if I had had some more clear indications from OpenEdge of exactly what the problem was, but I finally decided to go and try the connection from Java.

The Bible

Before I started on the Java example, I decided that I should probably test that I could even make this happen in .NET. Even that is no small feat. The examples that come with the Exchange SDK are about as useful as an udder on a bull, and I eventually decided to bite the bullet and look for a book on the subject.

I normally avoid Microsoft Press books because, more often than not, they simply regurgitate the standard Microsoft examples and contribute very little additional value. That is not the case with Inside Microsoft Exchange 2007 Web Services. Authors David Sterling, Benjamin Spain, Michael Mainer, Mark Taylor, and Huw Upshall have done an outstanding job of creating a book that really gets to the heart of the technology. They are all members of the Exchange development team, and it shows. The book provides enough background information that it is essential reading for anyone trying to do this stuff.

Java to EWS

Armed with my new reference, I now set about building a Java client that could create an appointment in Exchange, figuring that I could expose the Java client as a Web Service that OpenEdge could consume.

Now, before you throw your toys out your crib and tell me that that is a very expensive way to do things from a performance point of view, let me just clarify that I would have ended up going down this road eventually, anyway. Ultimately, the solution will be exposed as a service on an ESB, and that ESB is likely to be based on a JMS MOM, like Apache ActiveMQ, or SonicMQ. So this is not a wasted exercise. The reason that I need this on an ESB is that the events that flow in each direction need to be available to other services, too.

Java to EWS was not a lot easier to get working. I still had to work around the WSDL not having a "service" node issue, but Java gives you finer-grained control of the parsing of the WSDL. Moreover, in my initial implementation, I did not use JAX-WS or any other WebServices technologies like Axis. I simply used an HTTP request to the WebService to get through the issues. This made life a whole lot easier.

The SSL problem reared its ugly head again, but this time I quickly learned 2 things:

Exchange creates a default certificate for you when you install it. Either you have to install the certificate as a trusted certificate in the certificate store, or you have to set up a certificate authority on your Windows Server and build signed certificates. I went with the former idea and that resolved the first part of the SSL problem.
Exchange is installed to support only NTLM authentication, but NTLM authentication is a pain to work with from non-Microsoft technologies. You can change the settings of the EWS virtual directory on the Microsoft IIS Server that runs on the CAS to support basic and digest authentication. For testing, I set this up as basic, but I now have digest working. I will probably go back and revisit NTLM again later, but it is very hard to find any information to help with solving problems, so I may simply require digest authentication.

Once I got past these issues, creating the calendar item was actually very simple.

OpenEdge through the Java WebService

The next step was to take what I had created and expose it as a Java Web Service. I went with Glassfish V3 as the application server just to test this and in about a half hour I had the Web Service up and running and soapUI was creating appointments through the Java Web Service without any problems.

I then went back to OpenEdge and used its WSDL analyzer to analyze the WSDL. It worked really well. I then built a quick client that called the Java Web Service from OpenEdge and it worked first time.

Next Steps

Although this is now a working prototype, it is exactly that – a prototype. There are several pieces that still have to be proven:

I need to subscribe to change updates from Exchange.
I need to get to updates for all sales person mailboxes. This means I need to use Exchange Impersonation and that has a number of security risks associated with it.
I need to make sure these updates get through to OpenEdge efficiently which means I need a test harness that can generate several thousand appointments a minute and have them update the OpenEdge AppServer without breaking it.

Conclusion

Having said that, I am really excited and very optimistic about how this is turning out. I have learned more about Windows Server, Microsoft Exchange, Active Directory, Exchange Web Services, OpenEdge Web Services, and Java Web Services than I had ever imagined possible.

In the next few weeks I will be providing an update on where I finally ended up with this from an architectural point of view.

OpenEdge Dynamic OpenClient Java Example

Bruce Gruenbaum — Tue, 10 Nov 2009 02:27:48 +0000

In a previous post, I said I would post an example that demonstrates the use of the OpenEdge Dynamic OpenClient. Well here is the Java version of it.

It took me a while to get this done, not so much because of the code, but because the documentation that accompanies it is pretty extensive and I have been in the middle of some significant career changes.

This post is long primarily because I walk through the highlights of the what the code does and how this bolsters what I was saying in my earlier post.

Inside the zip file are 2 zip files and 3 PDF files:

DynamicOpenClientJar.zip is the compiled version of the code. It contains a jar file, the input.xml, a batch file, and the Progress source code.
DynamicOpenClientJar.pdf contains setup instructions to set it up and run it.
DynamicOpenClientSrc.zip is the source code.
DynamicOpenClientSrc.pdf contains the instructions that that tells you how to set the source code up to work with Eclipse.
DynamicOpenClient.pdf is a detailed tour of the code and describes how it works.

This is Sample Code

Before you dig into the code and say "Wow!! This code sucks!!" it is important to understand that this is sample code. There are a lot of things about it that are ugly. Some of the object model needs serious refining. There are cleaner ways of doing the XML parsing and it is nowhere near as defensive as it should be. There are those who believe that bad code samples lead to bad code in production, but there is another line of thought that goes that if you don't recognize those bad examples, you should not be coding to start with. The point with this code is that it demonstrates concepts and that's what it was intended to do.

Code Limitations

For the sake of brevity, I have deliberately not covered stuff like supporting supporting ProDataSets and all of the different Progress data types. I have covered the most common ones and temp-tables. ProDataSets are really an extension of temp-tables and you can feel free to go and look at the code in a generated proxy to see how they work. I also stopped short of supporting the building of temp-tables from an input XML file. The extra code to do this did not seem like it would really add that much to the example. This code example also does not support persistent procedures because they are a lot more complicated than this code will allow.

Set it up and try it

Having said all of that, I wanted to spend some time in this post elaborating on the comments that I made in the earlier post about why Dynamic OpenClient is such a powerful technology, using the example code that I am presenting here. So I am going to assume for the rest of this article that you have at least set up and run the example and that you have the source code accessible to follow along.

How it Works

The Dynamic Open Client code takes an XML file and parses it for AppServer connection parameters, R-Code signatures and calls that need to be made to the Progress/OpenEdge R-Code files. The sequence diagram at right (click on it for a full-size image) shows the process.

Once the XML file has been parsed, the utility executes each of the calls and serializes the results of the call's execution to an internal XML document. Once all the calls have been executed and serialized, the results are written out to another XML file that is specified in the input.xml file. More information on the XML file's format is in the DynamicOpenClient.pdf file.

The Main Components

To understand how this utility works, you need to understand the core of the class model. So let me start by dividing the class model into its major areas.

You can pretty much ignore the com.tsg.common project. It contains an interface - INamedItem – used by a few classes that simply has a getName() method declaration. INamedItem is implemented by some of the classes in the com.tsg.dynamicopenclient project. This interface is specifically used for objects that may be stored in a com.tsg.common.collections.OrdinalList – a specialized list that is ordinal like an ArrayList, but expects uniquely named items in the list. OrdinalList is used for things like field lists and parameter lists where the fields need to be in a specific order, but the names have to be unique.

The com.tsg.dynamicopenclient project contains three packages:

com.tsg.dynamicopenclient - This package contains the code that does most of the work;
com.tsg.dynamicopenclient.data - This package contains the class hierarchy for parameter and other data element definitions;
com.tsg.dynamicopenclient.valueholders - This package contains the class hierarchy for all the value holders that are used to contain the values of input and output parameters.

Data Types

One of the most important aspects of the communication between the OpenClient and the AppServer, is the mapping of data types from Java to Progress. To make sure that this is done consistently, there is a DataType enumeration in com.tsg.dynamicopenclient. If you have not worked with Java 5, enumerations may be a new concept to you and this one is particularly more so than others. An enumeration is a great way to deal with hard-coded constants and it is particularly useful where a map of constants is necessary. In this particular case we have to map an OpenClient numeric constant to a string representation of the data type, a Java class that can be used to map to that type and a valueholder class that can be used to contain the Java type. The following definition of the INTEGER data type demonstrates the mechanism:

INTEGER
  (Parameter.PRO_INTEGER, // Progress numeric constant
   Parameter.PRO_INTEGER_NAME, // Progress character constant
   "java.lang.Integer", // Java class to be used to map
   "com.tsg.dynamicopenclient.valueholders.IntegerValueHolder"), // Dynamic OpenClient valueholder

The DataType enumeration has several methods associated with it to retrieve a constant based on the string name of the data type or to obtain the Java class, value holder class or Progress data type constant. It also contains code that will instantiate an object of either the Java class to contain the data type or the value holder. This enumeration truly is at the core of the code.

Data Elements

The com.tsg.dynamicopenclient.data package contains a set of classes that define the behavior of data elements. A data element is any item that has a name and a datatype associated with it. Thus, fields and parameters are all data elements. Data elements only store the definition of the element. They do not store the value. The value is stored in a value holder. The abstract base DataElement class contains all the behavior for parsing parameter and field definition information from an XML node.

The Parameter class extends it by providing support for parameter modes (INPUT, INPUT-OUTPUT and OUTPUT) which are mapped in an enumeration called ParameterMode in com.tsg.dynamicopenclient.

The TempTableParameter class extends the Parameter class further by providing support for parsing temp-table definitions from the source code and supporting the construction of ProDataGraphMetaData objects that are used to build the ProDataGraph to transport temp-tables between AppServer and client.

Value Holders

The com.tsg.dynamicopenclient.valueholders package contains a set of classes and an interface that are used to standardize the way that data types are dealt with. Looking at the class model, the value holder hierarchy is to the right of the diagram. This hierarchy leverages the concept of generics that was introduced in Java 5.

The IValueHolder interface is the base of the hierarchy and the ValueHolder abstract class contains the generic behavior for value holders. These classes provide support for the Progress UNKNOWN value as well as reasonable default values that match the Progress defaults for its data types.

Value holders are used to set the value of input parameters and to contain the values of output parameters.

DynamicOpenClient Class

The DynamicOpenClient class in com.tsg.dynamicopenclient contains the main() method - the entry point to the utility. It starts out by parsing the parameters to determine where the XML file is that contains the signatures and input parameters for the calls. The input.xml file's structure is detailed in the DynamicOpenClient.pdf file that accompanies the source code. In essence, though, there are 4 types of nodes that are important:

dynamicoc – This is the root node for the document and it contains an OutputFile attribute that has the name of the file that should be used for the results of the calls.
appserver – This node contains the signature and call nodes and a set of attributes that define the connection parameters for the AppServer.
signature – This node contains the signature of a call that will be executed against the AppServer. The signature node may contain a set of parameter nodes which will be all of the parameters for a procedure. These parameters must be specified in the order that they are specified in the ABL source code. If the parameter's DataType attribute is TABLE, the node will contain child nodes that are the field and indexes for the table.
call – This node contains a list of calls that need to be executed against the AppServer. Each of them should refer (by means of the Procedure attribute) to a previously specified signature. call Nodes may contain subordinate parametervalue nodes that reference a parameter in the signature by name and specify the value to be used as the input value for that parameter.

The following code extract comes from the main() method.

//Extracted from main() method.
//

//Instantiate the CallManager
CallManager manager = CallManager.getInstance();

//Parse the XML file.
try {
	parse(fileName);
}
catch (Exception e) {
	System.out.println(e.getMessage());
	return;
}
.
.
.
Iterator iter = manager.iterator();
while (iter.hasNext()) {
	Call call = iter.next();  //Get the call
	call.executeCall();   //Execute the call
	outRoot.appendChild(call.serializeResults(outputDocument));
	iter.remove();
}
.
.
.
try {
	writeResults();  // Write the XML results file out to disk.
}
catch (Exception e) {
	System.out.println(e.getMessage());
	return;
}

Before doing anything else, the code instantiates the Singleton CallManager to hold onto all the data that will be parsed from the XML file.

The main() method calls the static parse() method with the name of the XML file to be parsed. This method parses the XML document and iterates through the four element nodes mentioned above. It also sets up a DOM document that will be used as the output document for the results of the calls.

For each element, it instantiates an object of the appropriate class and passes it the XML node that needs to be instantiated. This object is then added to the CallManager where it will be retrieved later.

//Extracted from parse() method.
//
Node elementNode = null;
while((elementNode = iterator.nextNode()) != null) { // Loop through elements

	// Handle root node.
	if (elementNode.getNodeName().equalsIgnoreCase(XMLStrings.INPUT_ROOT_NODE.toString())) { 	

		// Setup the output file.
		String attrValue = XMLStrings.getAttribute(elementNode, XMLStrings.OUTPUT_FILE_ATTR.toString());
		if (attrValue != null) {
			outputFileName = attrValue;
		}
	}

	// Handle appserver node
	if (elementNode.getNodeName().equalsIgnoreCase(XMLStrings.APPSERVER_NODE.toString()) ) {
		manager.setAppServerConnection(new AppServerConnection(elementNode));
	}

	// Handle signature node
	if (elementNode.getNodeName().equalsIgnoreCase(XMLStrings.SIGNATURE_NODE.toString()) ) {
		manager.setSignature(new Signature(elementNode));
	}

 	// Handle call node
	if (elementNode.getNodeName().equalsIgnoreCase(XMLStrings.CALL_NODE.toString()) ) {
		manager.addCall(new Call(elementNode)); 
 	}
}

The CallManager implements the Iterable interface for the Call class. Back in the main() method, the next thing we do is iterate through all of the Call objects in the CallManager and we execute the call. This is where the real work happens and a description of it is below. When execution is complete, the parameter values will have been stored in the Call object and a call to serializeResults() will return an XML node that can be appended to the output document. The Call object is then deleted from the list of calls to free up the memory that any temp-tables will occupy.

Once all the calls have been executed, a call is made to writeResults() which writes the output XML document to disk.

Executing the Call

A call is actually executed by a call to executeCall() on the Call object. The executeCall() method simply builds a parameter array, calls the runProcedure() method on the AppServerConnection object, builds the output parameter value set and handles any exceptions. The thing that makes this client dynamic is the code in buildParamArray() and buildOutputValues().

buildParamArray()

In order to make a call to the AppServer, the OpenClient needs a procedure name to call and a parameter array (com.progress.openABL.javaproxy.ParamArray) for the call. The parameter array needs to contain a full description of each parameter as well as the value to be set for the parameter. The Call object has a procedure name and it has the values of the input parameters. The name of the procedure maps to a Signature object which can be obtained from the CallManager and it contains the definitions of all of the parameters that are needed for the call. Thus, buildParamArray() obtains the Signature object and builds a ParamArray object with each of the parameter definitions in the Signature object. As it attempts to add each parameter to the ParamArray, it obtains a IValueHolder object to contain the parameter's input or output parameter value. It then checks for a corresponding input value in the Call object. If it finds a corresponding value it sets the input value in the ParamArray from this value. This is how the Call information in the XML file can specify the parameter values by name in any order and only specify the ones that have values associated with them.

//Extracted from buildParamArray() method.
//

CallManager manager = CallManager.getInstance();
Signature sig = manager.getSignature(getProcedureName()); //Get the signature for this call
if (sig == null) {
	throw new DynamicOpenClientException(...);
}
ParamArray params = new ParamArray(sig.getParameterCount()); //Instantiate a ParamArray object

int idx = 0;

// Iterate through all the parameters in the signature.
for (Parameter param : sig) { 

	// Instantiate an IValueHolder for this data type.
	IValueHolder valueHolder = DataType.getValueHolder(param.getDataType());
	paramValues.add(valueHolder); // Put the value holder into the paramValues list.

	// If we can find a parameter value that was set for this call,
	// set the value of the IValueHolder to the input value.
	if (inputValues.containsKey(param.getName())) { 
		valueHolder.setObject(inputValues.get(param.getName()));
	}

Now that the data we need is available (ie, the parameter type information and input values have been set), we can go about setting the values, but we need to treat the TABLE and TABLE-HANDLE types differently from the the other data types. The reason is that they need a ProDataGraph and its metadata. The code for setting the values is therefore as follows:

Object value = null;
switch(param.getDataType()) {
	case TABLE:
	case TABLE_HANDLE:
		TempTableParameter ttParam = (TempTableParameter) param;
		ProDataGraphValueHolder graphHolder = (ProDataGraphValueHolder) valueHolder;
		ProDataGraphMetaData metaData = null;
		switch(param.getMode()) {
			case OUTPUT:
				// For a table we need the table structure. TABLE_HANDLE needs null.
				if (param.getDataType() == DataType.TABLE) {
					metaData = ttParam.getProDataGraphMetaData();
				}
				break;
			default:
				value = graphHolder.getProDataGraph();
				metaData = graphHolder.getProDataGraph().getMetaData();
				break;
		}

		// Add a parameter to the ParamArray with the appropriate value.
		params.addParameter(idx, value, ttParam.getMode().getModeCode(),
			ttParam.getDataType().getTypeCode(),
			ttParam.getExtent(),
			metaData);
		break;
	default:
		// Now get the object from the IValueHolder that has the value in it.
		value = valueHolder.get(); 

		// Add a parameter to the ParamArray with the appropriate value.
		params.addParameter(idx, value, param.getMode().getModeCode(),
			param.getDataType().getTypeCode(),
			param.getExtent(),
			(ProDataGraphMetaData) null);
		break;
}

The outer switch in this code handles the difference between TABLE, TABLE-HANDLE and other parameters. So skip the first cases and look at the default case (the last 11 lines of this code block from the word "default:" on. All this code does is grab the object that is in ValueHolder and assign it to the value variable. It then adds a parameter specifying the ordinal position (idx), the value, the parameter mode (INPUT, OUTPUT or INPUT-OUTPUT), the Progress DataType Code from the DataType, the extent and a null for the ProDataGraphMetaData.

The TABLE and TABLE-HANDLE cases are slightly more complicated. For a TABLE or TABLE-HANDLE if the parameter is either an INPUT or INPUT-OUTPUT parameter, the value has to be set to the ProDataGraph that will be used to contain the return value. The ProDataGraphMetaData needs to be specified to supply the table structure. In the case of an OUTPUT parameter, a TABLE parameter requires ProDataGraphMetaData but null in the value parameter and in the case of a TABLE-HANDLE, both the metadata and the value need to be null. In both cases, the Parameter object will have been specialized as a TempTableParameter object so that it could build the temp-table structure.

Once the parameters have all been added to the ParamArray, it is returned to the caller (executeCall) and is now ready to invoke the call.

runProcedure()

The runProcedure() method call is run on the AppServerConnection object that is contained in the CallManager. The AppServerConnection class performs 2 functions:

It stores the AppServer connection parameter values; and
It executes calls against the AppServer.

The first thing runProcedure() does is call obtainAppObject(). The OpenAppObject class provides access to AppServer connections from the connection pool and supports all communication with the AppServer. The obtainAppObject() call instantiates a Connection object and then an OpenAppObject if they have not already been instantiated. This establishes the connection to the AppServer.

Once the connection is established, the runProc() method on the OpenAppObject is run with the ParamArray that was passed in and any exceptions are thrown back to the caller. If this code completes successfully, the AppServer call has succeeded.

buildOutputValues()

After the call has completed, executeCall() calls buildOutputValues() with the ParamArray that was used in runProcedure().

//Extracted from buildOutputValues()
//
returnValue = params.getProcReturnString();
if (returnValue == null) {
	returnValue = "null/Unknown (?)";
}
int idx = 0;
for (Parameter param : sig) { // Loop through each parameter in the signature.
	switch (param.getMode()) { // Check the parameter mode.
		case INPUT_OUTPUT:
		case OUTPUT:

			// Get the value of the parameter from the paramArray.
			Object outputValue = params.getOutputParameter(idx); 

			// Set the value of the parameter in our internal array to be the result
			paramValues.get(idx).setObject(outputValue);
	}
	idx++;
}

The buildOutputValues() method does 2 things:

It obtains the value of the RETURN-VALUE string from Progress; and
It loops through all OUTPUT or INPUT-OUTPUT parameters and sets the value in the IValueHolder from the value in the ParamArray.

The output parameter values are thus available to the Call object for serialization.

Why this is so powerful

That is the crux of the code in the Dynamic OpenClient Sample Utility. It's not really that complicated and although this is a crude example, it does show the relative ease with which one can build a client that invokes calls for which all signature information is obtained at runtime.

If you work through the associated document that contains a description of running the utility, you will notice how easy it is to edit the ABL code and change temp-table structures or any other signature related information, modify the corresponding signature in the XML file and re-run the utility without changing the Java code.

If you go ahead and automate the generation of the XML signatures into a post-compile operation and post it on a web-server or some other centralized resource, there is never a need to change the Java code unless you discover a bug in it or Progress adds a data type.

There's another interesting side effect to this. It would not take much effort beyond this to build an automated mechanism for dynamic WSDL generation so that you dynamically expose any ABL procedure as a WebService. Taking it further, exposing it as an EJB or J2EE container or a JMS endpoint would be fairly simple too and would mean very little would ever need to be changed in the plumbing between the client and the server. You see, as far as I am concerned, everything related to ProxyGen is plumbing that should never require a recompile unless the underpinnings change. ProxyGen forces that each time a temp-table definition is changed.

Hopefully this example is useful. I am planning a .NET equivalent later on.

OpenEdge Dynamic OpenClient

Bruce Gruenbaum — Wed, 29 Jul 2009 20:06:47 +0000

I forget how long ago it is that the OpenEdge OpenClient was released. It must be easily close on 10 years, but I am willing to accept that may be an exaggeration. OpenClient is a cool technology. For the uninitiated, the OpenClient is a very powerful library of code and a set of tools that allows .NET, Java and WebService access to the OpenEdge AppServer. It allows you to build your business logic in the OpenEdge ABL and access it from whatever client you choose.

In order for the OpenClient to be able to connect to the AppServer, make a call and return the output parameters, there is a certain amount of code that you need on the client (Java or .NET) that wraps the OpenClient library. In an effort to simplify the process of writing that code, OpenEdge includes a utility called "ProxyGen" that parses the OpenEdge ABL code and generates a proxy class that wraps all of the code needed and exposes it as a simple method call. Once you have generated the proxy, you simply include the generated class in your code (either as a .jar or a .NET assembly, or even directly as source code) and you can make whatever call you need.

The problem with ProxyGen is that you have to regenerate the proxy and potentially rebuild the client each time the ABL procedure changes its signature. That would be okay if the ProDataset and Temp-Table definitions were not part of the signature, but they are and adding a field to a temp-table should not cause a proxy to break.

"Hang on, Bruce," I can hear you say, "Why is the signature of the temp-table not a change to the signature to the API?" I'm not saying that it isn't, but I am saying that it is far more difficult to avoid making this mistake. Here's why:

Typically, temp-tables that form part of the in-memory data model are defined in include files and then used in procedures and classes as in the attached diagram.

Let's assume that Procedure1.p and Procedure2.p are deployed to the AppServer and that Procedure1.p accepts the temp-table as an input parameter and Procedure2.p uses it as an input-output parameter.

As a result of a necessary change to the in-memory data model, a field or two are added to the temp-table definition for the purposes of the Class1.cls and Class2.cls. As part of the build, the code is recompiled and the new code deployment to the AppServer is completed successfully. Note that there has been no planned change to the Procedure1.p and Procedure2.p.

The problem that we now face is that the proxies that were generated for the Procedure1.p and Procedure2.p are now invalid. The temp-table definition is no longer correct and a run-time error will now occur for the client when these procedures are called.

Another problem with proxies is the complexity associated with unit-testing and verifying each individual proxy. Each time a proxy is changed, each of the unit tests for each call in the proxy has to be rerun and retested. There is no way to generically test them to verify the Java/.NET code.

In one of the more recent versions of OpenClient, the API that the OpenClient uses is documented so that it is now possible to dynamically construct the proxy calls at run-time. The overarching benefit in this lies in the ability to now define the temp-table definitions on the fly. The Java/.NET code can now define the temp-table dynamically at run-time so that if a change is made to the definition, the client can deal with the change with no impact on the OpenClient source code.

This functionality is, in fact, so powerful that the entire signature, including all parameter definitions, can be discovered and defined at run-time.

This concept becomes even more interesting as you consider using it for WebServices. ProxyGen can generate the WSDL for a procedure that is to be exposed as a WebService, although it suffers from the same issues if the signature changes. The dynamic OpenClient API makes it possible to dynamically generate the WSDL at run-time from the procedure. Obviously there are overheads associated with auto-discovering a procedure's signature. It makes sense, therefore, to build out this functionality so that the signature discovery is done as seldom as possible.

In a future article I will provide some code samples that show you how to use the Dynamic OpenClient.

UPDATE: I have subsequently made this code sample available. You can refer to it here.