If this recession has done one thing, it has forced companies in corporate America to take a long, hard look at their relationships with their customers, and many companies are realizing that they need to be far more proactive about their relationships. 

But how do companies go about finding that "special sauce?" And more importantly, what can they do to maintain it?

While I don't believe for a second that all the special sauce in the world comes from technology, technology can often be at the very heart of making that special sauce, especially when we are talking about building customer relationships. At the heart of it is showing the customer that you understand them, that you care about them, and that you want to retain them as a customer.

Technology can be a huge support in this area, and that's where research and innovation come in. Using technology to build better business relationships is where the focus of technical growth is, and learning how to apply technology to do so is where research and innovation come in.

How to look at this

When I started work on the Microsoft Exchange Integration project, my first goal was to understand what the business was trying to do. In this case, it was keep better record of appointments that were set up with the customer, and monitor changes made to those appointments to ensure the best customer experience possible – a really simple requirement.

Once I knew what the requirement was, the next step was determining a) how to provide it, and b) what else I could do. In other words, I stepped back a level and looked at the big picture to strategically improve the interaction between sales and prospects. Rather than just answering the immediate requirement, I decided to look at how I could improve the overall experience.

Among the things I immediately realized was that not only could I provide a way to allow users to create appointments in Exchange, but I could also keep those appointments up to date on the database side. Not only that, I could even go as far as to allow users to query their own availability from within the application.

My next step was to figure out what I could really do with the technology, and this is where the real research happened.

The real research

Research is not just a technology thing. As a geek, I am inclined to look at the code and say, "Wow, that's cool." But realistically, an Enterprise Architect's job is to look at the business and understand how to improve the business by applying technology, and the first step is to understand the business, not the technology. So my first step was to walk through with the customer what his existing process looked like and where he wanted to optimize it. In other words, the real research was into understanding what he is trying to accomplish by improving his business process.

Sometimes when you are buried in a process on a day-to-day basis, it is really hard to recognize anything but the pain of the process. The reaction is to fix the pain, rather than the process. What often happens is that an outsider is able to look at the process and say "Why do you do that?" and make the process owner re-evaluate the process completely. Many times, the process is the way it is because it has always been like that.

So the real research is about finding a way to simplify and even eradicate process. What people often fail to grasp is that processes change over time. As systems become more automated, there is less and less need for the duplication of effort that goes into some processes.

Looking back at the Microsoft Exchange Integration example, not only does the process owner need the ability to monitor the appointment changes in terms of how often an appointment is changed, but he also needs to be notified when the number of appointment changes is exceeding acceptable thresholds, or (and this was not factored in initially) too much time has elapsed between the initial contact and the expected follow-up. Remember – we are dealing with a prospect. Prospects go cold if they are not reminded of their interest.

That last point made me think of something else. There are times when a prospect has to postpone appointments because of business pressures. Heck, I blew off a guy last week because I am so snowed at work there is not time to see him, but that does not mean I am not interested in his product. Right now is just not a good time. Of course, I am likely to forget about him, so it is in his best interest to follow up with an e-mail or something that reminds me that I was interested in his product. Don't pressure me for an appointment. Just remind me about the product and its capabilities – something like "Our product just helped this customer achieve this business objective. You may be interested in reading about it."

Notice that these are not part of the initial requirements, but based on my own experience I can think of about 20 different things that the process owner could do to maintain the prospect's interest and further the sale.

So as I was researching this, it occurred to me that the process owner would benefit from automatically being able to pass on useful information about his product to his prospects that have not had any communication for a period of time. That nudge may trigger deeper interest. This made me realize that the process owner could really benefit from a knowledge-base type application of customer experiences that could be formatted and e-mailed.  

Another thing to notice about this is that before I ever looked at the technology that I was about to use, I first considered the business problems I was trying to solve. Thoroughly researching the business needs is absolutely critical to providing a valuable solution.

Takeaway #1: The thing to take away from this is that, as an Enterprise Architect, researching the business process, how it works, and what executive management's objectives are, is way more important than the technology research. If you do a thorough job of the business process research, it allows you to focus the technology research.

Researching the technology

Of course, as a geek, there is nothing that I love more than trying something completely new. I can spend hours messing with technology and trying new stuff. In fact, one of my greatest weaknesses is that I enjoy doing new stuff so much that I have to really force myself to finish things off. So when I get to mess with technology for the fun of it, I am in my element.

As I said, though, the important thing with technology research is to have a goal in mind. Why are you looking at the technology and what do you think you can accomplish by messing with it? What real business problem can you solve with it?

With the Microsoft Exchange integration project, a lot of those things were very clear from the minute I had the business problems clear in my head, so the next step was to understand the technology. My first step was to understand what Microsoft provided and it very quickly became clear that the Exchange WebServices (EWS) API was their new direction and I had to take this direction. Cool. Now I had to understand what EWS could do. That was mostly a reading exercise to understand it from Microsoft's point of view, and their is a very informative forum on Microsoft's site where you can get an idea of some of the problems people have faced.

Of course, Microsoft does not have all (not even most) of the answers. Here's where I cannot possibly imagine life before Google. Most of the posts on the Microsoft forums are by people who have spent way too much time drinking the Microsoft Kool-Aid, not least of which are the people over at the MSDN "All about interop" blog who make this inane statement: "While I love .NET, not everyone has seen the light! and some poor developers are relegated to using Java even for new applications."

OK, you brain-dead numb-skull! Let's see you try any serious Unix development in .NET. Yeah, Mono is there, but Microsoft doesn't even support it. Unfortunately, for every brain-dead Microserf, there is an equally ritalin-deprived Javazoid who is so bigoted that he cannot see the forest for the trees either. There is absolutely no place for technology bigotry like that in real Enterprise Architecture where the technology is definitely secondary to the business needs. 

I knew most of my work was going to be Unix-based and I knew that I was not going to find anything that was OpenEdge-related, so Java was the obvious other platform to use to talk to Exchange. A quick Google search of "exchange web services java" turned up a number of hits, but the most valuable one was undoubtedly Reid Miller's blog.  He did a really good job of walking through the process of connecting to EWS from Java, and without it I would probably have torn my hair from my skull in chunks.

Google also referred me to a web-site that sells two products called EWSJ and JEC. These are Java-based libraries, one of which makes a connection to Exchange Web Services. The libraries look very promising, but there were several reasons why I did not like their solution, not least of which is the fact that I need to connect to Exchange Web Services from OpenEdge, so this was not going to solve my problem.

One thing that I learned that has proven to be unequivocally true is that NTLMv2 authentication, which replaces NTLM authentication on Microsoft Server 2008 R2, is really tough to do in Java. From what I can make out, there is a way to do it if you buy a third-party library, but I have not proven that the library works and I am not sure that it is warranted. This helped me make a very important decision long before I started writing any code. Instead of fighting the Microsoft authentication battle, I would rather sort that out later and use digest authentication from Java to EWS. 

Another very valuable thing that I did was buy the book. As I said in my original post on the Exchange Web Services project, Dave Sterling, et al's book on Microsoft Exchange Web Services is invaluable and it taught me so much stuff in so little time that it was worth every cent of the $40 that it cost me.

A clever man learns from his mistakes. A wise man learns from others' mistakes. - (My grandfather, Eric Frank Brading – 1926-2008, one of the wisest men I've ever known)

Takeaway #2: Spending a few hours digging into what others have done can save you a ton of trouble.

At last – Try it out

My last step was to try out the code. Within 2 hours I had a connection to EWS working from Java. An hour later, I had it exposed as a WebService, running on a CentOS box in a Glassfish V3 application server. Ten minutes after that, I had OpenEdge calling the Java WebService to talk to EWS. So four hours of experimentation got me the working prototype to create appointments in Exchange.

Note that my focus was to meet the business requirements in what I was doing with the prototype. I tested to prove that I was able to do the main thing that my customer was asking for. Sure, the code is ugly, but who cares? It's only for testing.

The next step was to prove that I could listen for messages from Exchange. In other words, I needed to get the notification API working. This was a little harder because I was hoping to get the communication to go through an OpenEdge AppServer. EWS expects a specific format and I could not get my OpenEdge service to meet EWS' requirements. This is where you have to make some early decisions and here's another takeaway:

 Takeaway #3: Fail early. If it's not critical to business success, don't waste your time on a perfect technical solution. Learn to compromise and come back to the imperfections later.

Absorb the results

Once I got the notification API working, I spent some time building some test code that would demonstrate performance. Now you may wonder why I was worrying about performance so early in the cycle. Here's why: Performance is the most misunderstood concept in the computer industry. Understanding latency and where the potential bottlenecks are early in the cycle will change the design of your application. Instead of building unnecessary complexity because of irrationally anticipated performance problems, hard, cold, data allows you to make a more sensible decision.

Another point to bear in mind is that as you experiment with it, you will learn a lot about the underlying technology. Save your results away because questions will crop up later that you can always go back and re-test, and this brings us to the next takeaway:

Takeaway #4: Your research is always going to lead to more questions than answers, and sometimes you will only think of the questions later. Make sure you keep all your research so you can always refer back to it.

For this reason, I check all my research code in to my source code repository and I comment it like I would commercial code so I can always go back and figure out what it taught me and how I was thinking when I wrote it.

And now for the innovation

As you absorb the results of the research, you will quickly see things that you can do with it to apply it to the business needs. One of the things that hit me very hard is that I can now synchronize the contact list in Exchange with the CRM system so that the contact list can be more accurate. I also realized that we could build a monitor for an e-mail address so that we can automate the processing of e-mails as they come in to things like the Sales alias. I could automatically create a prospect and generate the information a sales person needs to react to the prospect.

To really innovate, step back from the technology, look at the real business needs and figure out how you can do something that will radically change the business. For this project, that will come from exposing the whole API to a service-oriented architecture that is founded on a solid middleware solution. This will produce a mass of events for the business that can be used to make real-time business decisions based on very valuable real-time information.

So our last takeaway is:

Takeaway #5: When you are considering how to apply the technology, step back from the code and look at business problem again. Look at the objectives that the business has set and make sure that what you are going to recommend will deliver outstanding competitive advantage.

After all, that's the secret sauce.

As I said in my last post, I do not like to do work that could be automated. A large part of the work that should be automated is the work around the build process. More than anything else, successful software development depends on being able to produce a repeatable build process where the code that is built is thoroughly tested, installed and verified before it is considered stable. To get to the point of understanding how this all works, the hardware and network infrastructure is pretty important. So that's where I am going to start.

Hardware

The diagram at left shows the hardware configuration that I have in place. I have a wireless hardware router that is connected to my DSL line to the internet. Connected to that router are several Linksys hubs (there's only one shown in the diagram because it is the only one that bears upon the discussion), and a Netgear 10/100/1000 switch. Why do I have all of these?

Well I have several machines that are unrelated to my work that are connected to the network. All of our entertainment devices (DIRECTV set-top boxes, Playstation, Wii, and Blue-Ray players), my kids' notebooks and desktop PCs, and my wife's notebook are all connected to the network, some via wires and others wirelessly.

The Dell Inspiron Laptop and its companion Dell Dimension Desktop are two older machines (they're both over 5 years old) that I use for all kinds of testing and experimentation. I'm not afraid to replace and reinstall operating systems and software on these machines all the time. The Inspiron Laptop functions as my Linux server when I am presenting and I need a server away from home.

My HP Pavilion notebook is my workhorse workstation. All my blogging, development work, specs, and photography stuff (yes, I enjoy amateur photography in my copious spare time) all happen on my notebook.

One of the most useful and valuable purchase I ever made was the purchase of a LaCie 2Big Network drive. It is a network file store that hangs off the Netgear switch at 1Gb. This device contains all our backups, and acts as a file server for all the other devices on the network. I have a few 1TB external drives and once a week I plug in an external drive and back up the LaCie drive and store it off-site. All the other equipment backs up on to this device so we can always go back to it to restore.

The Dell PowerEdge T710 is a relatively new addition to the network. As I mentioned in my previous post, it hosts the virtual machines that I need to get most of my work done. It has 4 physical network cards that are all connected to the Netgear switch. 

When I first got the PowerEdge I was worried about how much power it would consume. Surprisingly, it is more cost-efficient than the Dell Dimension Desktop, which is why I was able to free up the Desktop. Until fairly recently, the Desktop performed the role of DNS, DHCP server, NIS server and DMZ. That has now changed. The DMZ has now moved onto the PowerEdge and runs in a separate VM. The DNS, DHCP and NIS server now also run in a separate VM and the firewalls are set up appropriately. My network is also segmented so that the personal network and the work network are completely separate.

Virtualization

I have been looking at virtualization for quite a while. When I worked for my last employer, we had a couple of Sun servers that were running OpenEdge AppServers, databases and other software that I was using to diagnose issues with their OpenEdge to Java configuration. I was surprised at how the system coped with the load that I threw at it, even though it was running off the same processors.

When I started work on the Microsoft Exchange Integration project, I knew I was going to need at least 2 Windows 2008 Servers and a couple of Linux boxes. I spent some time with a colleague of mine and discussed the issue and he suggested looking into VMWare again. I had used VMWare Fusion on my Apple Powerbook at my previous employer, but had not really understood how powerful this product really is.

My colleague told me about VMWare ESXi, which is a free version of VMWare that will run on a single machine and allow you to administer that machine as an isolated server. We spec'ed out what we thought I would need for the work I am doing and I ordered the PowerEdge accordingly. When it arrived, I installed VMWare ESXi on it and it has been set up and running now for almost 2 months with no down time.

What has stunned me is how efficiently this machine runs. At any point in time, I have at least 8 machines (3 of which are Windows Servers) running on this machine. The thing that I completely missed out on is how much time the machine spends doing nothing so that there is a lot of overlapping processor downtime that can be exploited. You can easily over-provision this machine and still have it perform very well. 

I'm going to need to spend some money on another 16GB of memory and another 2 TB of disk space, but that is a very simple upgrade for this box. 

The Big Score with Virtualization

The thing that I had not considered about virtualization that has been the biggest score is the fact that everything around the virtual machine management can be scripted. Not only that, it can be scripted from a Linux virtual machine running on the hardware that is running the virtual machine!

Now the reason this is really cool is because of the automation of the build process, and this is where the whole infrastructure discussion culminates.

As a software engineer, I live by the following rule:

 The software does not work until it is packaged, installed, and run on a(set of) machine(s) that is(are) not your development environment.

In other words, the fact that the code works on my notebook does not mean that it works in production. To prove that it does, I have to check it in, have an automated build kick off and build the code, have the automated unit tests run, deploy the code to the test environment, have it run there, and have the tests all pass. Finally, and this is really important for the Microsoft Exchange Integration project, I need to understand how all of this is likely to perform under load.

Building and Installing

With virtualization, I am able to create snapshots of machines, power them up by means of a script, install new software by means of a script, extract the results, power down the machines, revert back to the snapshot, and start again whenever I want to.

By means of one script, I am able to validate how successfully the build ran, and as it is a cron job that runs off my development server on a nightly basis, I have a good build and test report first thing every morning when I get up.

Stress Testing

Moreover, the test is able to use my quiet time to run any stress tests that need to be run. Assuming the build is successful (95% of tests pass, and none of the critical tests fail), the build is stored to the LaCie drive. At 8:00am every morning, the stress test kicks off because this is when the network is likely to be the quietest and the others in the house will not be affected. The script powers down any machines that are not needed during the stress test, starts the stress test machines, runs the stress tests during which time the processors are pegged at 95%, and keeps a log of how the stress tests ran.

At the end of the test (at 4:00pm) it e-mails a report for the tests and reverts to the snapshots for each of the virtual machines after it powers them down. Before it reverts to the snapshot, it copies the virtual machines across to the LaCie drive so that I can always restore the tests and see what the logs looked like if there is any question.

The cool thing about the test ending at 4:00pm is that I normally get the e-mail for stress test on my Blackberry as I am driving out of the parking garage at work, so I know what I am in for when I get home that evening.

The Value of Laziness

The real value of being lazy is that because I won't do repetitive things manually, I have put together a fairly stable build configuration that allows me to prove the code thoroughly before it ever makes it out the door. There are several important things that this process relies on, though.

I have to be ruthless about building tests for each of the new features that I build. This does not mean I believe in Test Driven Development – I don't. Test Driven Development is about building tests and then coding for the tests. I believe in building the code and then figuring out how to break it, and trust me, I am good at breaking my own code.

I have to make sure I add additional load to stress test additional features where I can. This can be really tedious because I spend a lot of time writing code that generates data.

Finally, I couldn't do this without software like Hudson CI, JUnit, NUnit, Apache Ant, NAnt, Apache Maven and most of all, VMWare.

Actually, now is a really good time to have this conversation because with the work I am doing on the Exchange Web Service code, I have just finished revamping my internal infrastructure to support the equipment and software I need to do the job. So this is going to be a two-part article. In this part, I'll tell you about the software development components that I use. In the next part, I'll tell you about the infrastructure components. The problem is that you need to understand some of the details of why, so I'm going to start with a little background.

Background

There are a few basic facts about me that effect everything else:

1. I am not rich. I have to rely on the most cost-efficient solutions I can. That means that I prefer open source/free software where it makes sense.
2. I am not cheap. I recognize that building software that is commercial grade requires a serious commitment to quality. That means a serious commitment to the development process, which, in turn, requires things like solid designs, automated builds and testing, and extremely good discipline.
3. I am the worst perfectionist. A gentleman I used to work for back in South Africa, Jan Fernhout, used to tell me that I would way over-engineer things because I wanted a perfect design. That is true. I have to be very disciplined about remembering that I need to deliver something.
4. I hate mundane work. If at all possible, someone else is going to do the work for me, and I there is no way that I am going to manually do something that can be completely automated. If you make me do it, I will bleat like a sheep until you fix the problem – just ask some of my co-workers.

There are also a few facts about the development work that I do that affect everything, too:

1. I believe that Windows is for the desktop or as a departmental product and Unix is what the enterprise needs. No one has proven me wrong on this. For every example of a large Windows product installation, I can point you at hundreds of more successful, more cost-efficient, more stable, less administered Unix installations that just run.
2. I do a lot of work with both Unix and Windows.
3. Java is the most platform-independent language for back-end work.
4. .NET is the richest GUI experience you can have outside the browser on the Windows platform.
5. OpenEdge is a fact of my life. I have used it more than any other development platform. I have made a good living off it, and I love the product, warts (many of them) and all. OpenEdge is kinda like an old pair of shoes that fits your feet a certain way and you're always most comfortable wearing them, no matter how ugly they are.

Machines and Operating Systems

My desktop machine – and primary development workstation – is a Windows 7-based HP Pavilion Entertainment 64-bit laptop with 6GB of RAM and 450GB of disk space. It runs nothing but Windows, and I have a ton of software on it.

My primary server machine is a  Dell PowerEdge T710 with 2 4-Core Intel Xeon processors, 16GB of RAM and 2 500GB drives. It runs VMWare ESXi (see screenshot at right) with 20 different virtual machines on it, some constantly running and some not. I'll tell you more about this in the next post on infrastructure. There are 6 VMs, though, that are important to this discussion:

• A Windows Server 2008R2 primary domain controller (pdc);
• A Windows Server 2008R2 build server (winbuild);
• A Windows 7 32-bit build machine (clientbuild);
• A CentOS-64 development server that hosts a whole lot of stuff (develop);
• A CentOS-64 build server (build64); and
• A CentOS-32 build server (build32) .

I then have 2 other machines that are both older machines that I use for various things. I'll talk about these in my infrastructure post, too.

Components of a Development Environment

As I said, I'm the worst perfectionist. So development starts with an idea, and ends when the code has been packaged, deployed, and installed on the end-user's machine, and they have completed user acceptance testing. In other words, it not only spans the entire Enterprise Architecture stack, but it also spans the Software Development Life-Cycle (SDLC). 

Now as much as I said I am not rich, there are two very expensive tool-sets that I have to buy:

1. Visual Studio Premium with MSDN. This one sticks in my throat every time I pay for it. The only reason I need the Premium subscription is for the Windows Server and Microsoft Exchange Server licenses. The more than $5500 that this product costs is very hard to justify, and the $2300 a year charge for renewing the subscription is as bad as paying tax to the IRS. I am a firm believer that development tools should be free. If you want me to develop products for your platform, give me the tools I need to do it.
2. PSDN Professional. Although I also believe that Progress should provide its tools for free, I am less resentful about it, probably because I used to work for the company. I still think the $3,000 that we pay for PSDN is excessive, but there you have it. My guess is there would be more people developing Progress-based applications if the development tools were given away free.

I need the MSDN license for the operating system and server licenses that I am using for Exchange Server. For my non-Microsoft-based development, I rely heavily on the Java and LAMP (Linux-Apache-MySQL-PHP) stack. Because I have some experience with PHP, I try as far as possible to use Open Source tools that are based on PHP rather than Perl. I don't know Perl and I don't particularly want to spend the time on it.

So starting from concept, here are the tools that I use to get my stuff done:

•  Sparx System's Enterprise Architect (EA) for all my modeling (not just diagrams, models). The product cost me about $300 for the original license, which included the MDG technology for Eclipse integration, and I use it as a design tool from the conception of an idea all the way through delivery. It contains the business process models, the requirements, and all the UML diagrams and test cases. I am not a fan of model-to-code driven development. I think the fact that we have been looking at the equivalent of CASE tools for the last 20 years shows that there will always be those cases where the code cannot be automatically generated for you. Two of EA's strongest suites are the traceability it provides from requirements to test case, and the incredibly powerful document generation facility. I literally generate all my documents from Enterprise Architect.
• Collabnet Subversion is used as my Source Code Control System. Everything goes into Subversion, including my models, source code, document artifacts, and even my research code. Subversion runs on my development server (develop) on CentOS. Subversion has clients for all kinds of platforms, so I have AnkhSVN plugged into Visual Studio to control my .NET code, I have Subclipse plugged into Eclipse for all my Java code, and I use TortoiseSVN at the operating system level to control documents and stuff that are outside of either development environment.
• Mantis is used as my issue tracking system. There is an Eclipse plugin for Mantis so I can get at it directly from my project workspaces in Eclipse.
• Hudson CI is used as my automated build and test environment. It runs on develop, but starts builds on build64, build32, winbuild, and clientbuild. I use both Apache Ant and Apache Maven to build Java code, and I use PCT to build OpenEdge code in Hudson. I use Nant to build .NET code.
• Unit testing is the scope of JUnit and NUnit although I am vehemently opposed to test-driven development. I think it is flawed and creates really lousy software.
• Eclipse is my Integrated Development Environment of choice. I have at least two installations of Eclipse at any one time. One is for non-Enterprise development (stuff that does not require Java EE) and the other is for Enterprise development (Java EE). The Eclipse screenshot at the top of this post shows my working environment for non-JavaEE work. From the screenshot, you will notice that I have OpenEdge running inside the Eclipse environmnent (Database Explorer is at the bottom, and there is an OpenEdge Editor open above it. You can also see an Enterprise Architect model in the right-hand pane, and my Hudson build status is in the left-hand pane. In the Eclipse screenshot at the bottom of this post that shows my working environment for JavaEE work, you will see the Mylyn task list at the bottom on the left, with the Mantis task repository open in the pane on the bottom right and a Mantis task open on the right. All of the tools that I have highlighted are integrated right into my Eclipse IDE in both environments. 
• Visual Studio is used only for development work that doesn't work in Eclipse. So most .NET/Windows-based development is done in Visual Studio. I have both Visual Studio 2008 and 2010 installed. 

All the tools I have discussed so far have plugins for Eclipse and, as the screenshots show, that is the configuration in which I use them.

A couple of other products that I use extensively that are not really integrated into the development environment per se, are:

• Beyond Compare is one of those tools that I just cannot imagine not having. It is a tool for doing advanced 3-way compares of files and it is, without reservation, the best diff tool that I know. It runs on both Windows and Linux, and I use it on both.
• WinZip is another tool that I use so much, I cannot imagine being without it. It is used for creating zip archives, but it can just as easily read and understand Linux tarballs and .jar and .war files.
• Nero is an outstanding tool for burning media, especially CD or DVD burning.

Most of my publishing stuff is done with either WordPress or Drupal. Like most of the other stuff I use that is web-based, these two are both PHP-based solutions. PHP is not a hard language to learn and it is very good at getting pretty sophisticated web apps up and running relatively quickly. WordPress and Drupal both have extension APIs that make it possible to build pretty sophisticated plugins for both in PHP – something I have done reasonably effectively.

Conclusion

Most of the tools I have mentioned above are freeware or open source or really, really cost-effective, and the good thing about them is that they lead you to build much better quality code than would otherwise be the case. If you are not looking at tools like these, I would highly recommend giving them a try.

Working without a data strategy is analogous to a company allowing each department and each person within each department to develop its own financial chart of accounts. This empowerment allows each person in the organization to choose his own numbering scheme. [...] Even to those of us who don't wear green eyed shades, the resulting chaos is obvious and easy to predict – (pg. 3, para 2).

What astounds me is how a risk that seems so blatantly obvious is completely missed by the majority of enterprises. In a recent study by Forrester Research, they found that 74% of over 400 companies surveyed view data strategy as critical or very important, but only 17% of them had a mature data strategy in place – (Topic Overview: Information Architecture by Gene Leganza, January 21, 2010, Forrester Research).

When you consider that most enterprises are outsourcing a substantial part of their core business systems, it is frightening that they do not have a strategy in place. The result is that each of their vendors defines their own view of the data and the enterprise loses control of what happens with their application infrastructure.

Scott Busse, in an article entitled Describing a Data Strategy to a Business Leader, called this uncontrolled, evolutionary data strategy a "waxy build-up" that leads to "higher costs, rigid processes, and a lack of insight into enterprise data" – (para 4). 

In this article we will briefly look at what Data Strategy is, and then focus on how data architectural integrity can be maintained in the Enterprise Architecture process.

The Scope of this Article

While the data architecture discipline owns the responsibility for the overall data strategy, I want to narrow data architecture for the purposes of this discussion. A formal, mature data strategy supports the enterprise by providing at least the following:

• Enterprise Data Model – The EDM is a model that defines all enterprise-level entities at a logical level, their relationships to each other, the life-cycle of each entity, what systems and services act upon it, and where the entity is used in the enterprise. It also defines the attributes of each entity and forms the basis for a common language for the enterprise.
• Canonical Model – The Canonical Model is a physical data model that is used by all applications and services to interact with each other throughout the enterprise. We'll talk more about the canonical model later.
• Metadata Management – Metadata is data that describes the entities and attributes in the Enterprise Data Model. So, for example, an account number may consist of 10 digits, of which the first 9 are a sequence number and the last digit is a check-digit. All of that information is metadata about the account number. This is a simple example, and in a large enterprise data model, there is a lot of metadata associated with the model.
• Data Quality – The GIGO principle (Garbage-In, Garbage-Out) is absolutely true about data and when strategic business decisions are being made based on invalid data, the results can be catastrophic. Ensuring that data integrity is maintained is a key part of any data strategy, and it has to be enforced in the data architecture as much as it is in the processes that govern the data.
• Data Governance – Managing and maintaining data from an operational point of view, ensuring that it is backed up, determining how long it is maintained in various stores, ensuring that the data stores perform well, and developing a high-availability and disaster-recovery policy all falls under the banner of data governance.
• Security and Compliance – Security and Compliance need to be closely aligned with the Data Governance component of a data strategy, but these functions are responsible for ensuring that legal compliance with things like SOX and PCI are taken care of. Security is responsible for ensuring data availability to the authorized channels while securing it from unauthorized ones. 

This is far more than I plan to focus on in this article. I want to talk about how you go about deriving the Enterprise Data Model components and the Canonical Model from the business process models that were created for the Exchange Integration Project that I blogged about a week or two ago.

The Business Process

The process at left deals with what happens when a sales person initiates a call with a prospect. The sales person initiates the call.

If the call does not succeed (i.e. the call fails, or the prospect is not available to take the call), the sales person documents the fact that the call was attempted in the system.

If the call goes through (i.e. the prospect is available for a conversation), a discussion is had and the sales person documents the conversation. Once the call is complete, the sales person documents the details of the call and determines the next steps to be taken, including any follow-up calls.

There is no question about the fact that I am way over-simplifying the business process in this discussion, but in order to keep this post as concise as possible, I need to do that. You can assume that some of the items that are noted as activities really need to be spelled out in more detail, and the business processes that are documented in the System swim-lane need a lot more information, but please cut me that slack… I need to keep this explanation short.

The Business Entities

One of the first things we can do with this model is define some of the business entities in this enterprise. In this process we have two entities who are really actors in the process:

• Prospect – The person who may be interested in goods and services from the enterprise; and
• Sales Person – The person responsible for pitching the sales of the goods and services to the prospect.

There are also several pieces of information that we care about that are flowing between these entities:

• Call – The fact that a call took place is important. We need to know the date and time of the call and how the call progressed;
• Discussion – The discussion that took place between the sales person and the prospect probably contains some very important information that would be useful to the enterprise if it were recorded; and
• Next Steps – The action that is to be taken as a result of the call could range from a follow-up call to the generation of a quote, or even an order.

So let's take these entities and model them to get an idea of what they look like.

 All of the entities that I references above are now part of this business entity model. We obviously have the prospect and sales person. We also have a call and a call next step. You'll note that the discussion got absorbed into the call as an element.

Now clearly, there is a lot more work that can be done to dig into each of these entities and refine them. Through an iterative cycle with the business, it is not hard to understand that a sales person is generally responsible for a prospect.

Moreover, a sales person is technically either an employee or contractor. That means that their information is available from other sources and in this context, they are simply performing a role in the business.

By the same token, a prospect may very well be an existing customer, in which case you want make sure that the information that you have for this prospect is stored for the customer. When the customer is a large organization, the prospect may, in fact, be a new department in the organization to which you have not sold before.

If a lead management system is present, the prospect may be an existing lead. So what is the difference between a prospect and a lead? Well a lead is a tip that you get about someone who may be interested in the product. A prospect is a lead for which the prospect-to-customer conversion process has already begun. 

A call is simply a way of interacting with a prospect. Other ways are by means of meetings, e-mails, letters, and other communication devices. Thus, the call is really an interaction.

Lastly, the call next step may be related to an appointment in the sales person's calendar, an order, or a quote.

So as you dig into each of these entities in more detail, it becomes clear that the model is significantly more complex and detailed than the business entity model we started defining above.

The point that is worth bearing in mind, though, is that these business entities were fairly easy to derive from the business process model that we discussed above. Although this process becomes more complex as the business processes are more complex, the same basic rules apply:

1. Each swim-lane is a major business entity;
2. Each line that crosses a swim-lane is a significant item of business data that needs to be tracked; and
3. By iteratively digging in to each of the business entities, it is fairly easy to identify numerous other affected entities.

Entity-Event Matrix

Once the business entities have been defined, it is important to understand the impact that each business process has on a business entity. To do this, I like to use a tool known as an Entity-Event Matrix. It is effectively a table with the list of entities across the top and the list of processes down the side. Each cell contains the actions that may be performed by that process on the entity. In the following table I have restricted my actions to create, read, update or delete, but in reality this becomes set of actions that specify the details of the changes made.

This matrix evolves as the model that it is associated with grows. Once the matrix has been completed for all business processes and aggregated together, any entity can be extracted from the matrix and a life-cycle for the entity can be produced.

At this point we have only discussed things from a business point of view. In a mature organization, the above deliverables are a function of the Business Architecture discipline (and I'm lumping Business Architecture and Business Analysis together).

The next step is to take these business entities and compare them against the Enterprise Data Model to determine whether there are any gaps.

The Enterprise Data Model

The terms "Enterprise Data Model," "Enterprise Conceptual Data Model," "Enterprise Logical Data Model," and "Canonical Model" are often bandied about in the industry somewhat interchangeably. So to make sure we are on the same page, I am going to define the Enterprise Data Model as follows:

An Enterprise Data Model is a logical model of the entities that exist across an enterprise, including their relationships, attributes and methods, as well as usage and value, and it is designed as a communication tool between all business, IT and external stakeholders.

There are several key things to bear in mind about the enterprise data model:

1. It is Enterprise-wide. That means that is the definitive guide to all entities that make the business work;
2. It is a logical model. That means that it does not have a physical manifestation, but it is used as the basis for defining the canonical model, which I will discuss a little later;
3. It is a communication device. That means that it is intended to help the business communicate with the business architects and all the other layers in IT. It effectively provides the common language that allows business units to properly communicate between themselves. It also serves as a reference point for external communication with partners, vendors, and outsourcing partners.
4. It is more than just an ER model. Beyond the entities and attributes that make it up, it also provides an impact analysis tool that allows the business to determine what systems are affecting different entities, where attributes are being realized, and what value data elements have to the business.

These are just four of the things that the Enterprise Data Model is; there are several others, but they are not germane to this discussion.

Once the business architects have done their due diligence, data architecture is engaged to validate the business entity model against the existing enterprise data model.

The model shown at left deliberately excludes attributes to simplify the diagram, but in this model, the data architect has taken the business entity model, mapped the entities and attributes to the enterprise entities and attributes, and identified any gaps in the enterprise data model. For these purposes, the calendar item entity is the only one that was missing.

In a mature organization, the enterprise data model is subject to a rigorous release cycle, much the same way as application software is.

Of course, the enterprise data model is not only affected by changes that are introduced by new business requirements. Often times, during the process of building components of the enterprise solution, additional entities and attributes are identified that should be included in the enterprise data model. Generally, this will trigger an architectural review of the enhancement as it is likely that other, more serious gaps exist. Nonetheless, it is very possible that additional entities and attributes can be introduced from sources outside business architecture. 

Once the enterprise data model changes are approved, focus shifts to the canonical model. 

The Canonical Model

Perhaps the most misunderstood, maligned concept in service-oriented architectures is the idea of a canonical model. Gregor Hohpe and Bobby Woolf spoke of the concept has a design pattern for enterprise integration in their book Enterprise Integration Patterns. While they were probably the first to document the pattern, there is nothing new about the idea. Many people have been using it for a very long time.

For example, about 12 years ago I was working on an EDI integration with a number of external parties. Each of the parties had a different standard and each of them needed data from one or more of the other parties. The simplest way to achieve that integration was to transform the data format from one party to a common format and then transform it to another format for the receiving party. This had nothing to do with SOA.

Actually, back as far as 1991, I was on a project that involved a conversion from several systems into a new system. Each of the old systems had data that contributed to the new system and all of the contributed data needed to be written in a single transaction to ensure referential integrity. The conclusion was to create a common model that could be addressed by all of the interested systems, and then transform data into that common model before moving it to the new system.

A canonical model is therefore a layer above all applications to which each application is able to transform their data to integrate with other systems.

In the context of our example, the canonical model is responsible for providing the transformation layer between the CRM application, the LDAP Server and the Microsoft Exchange Server. To create a calendar item in Microsoft Exchange Server, we need to draw data from the CRM application and the LDAP Server and then send it across to Microsoft Exchange in the correct format.

The CRM application has the Call Next Step data in a certain format. The LDAP server has directory information about the organizer and the attendees for the calendar item that needs to be created in Microsoft Exchange Server.

By creating a Calendar Item in the canonical model that conforms to a certain standard format, we can extract data from CRM and LDAP, transform it and store it in a canonical Calendar Item, ship the calendar item to the Microsoft Exchange Server, and transform it into the Exchange Web Service Item form that does not mean anything to anyone other than Exchange. Other systems that wish to make use of the Calendar Item can also perform their own transformations to their native formats.

More than just data, the canonical model also contains the methods that operate on canonical model objects. Moreover, the canonical model is often implemented as a relational model, an object-relational model, and an XML model. 

Persistence Models

Up until now, we have not dealt with persistence models at all – that is, models that are created to persist the data to a physical data store. As the canonical model effectively defines the entities and attributes for the enterprise, it can often be easily adapted for use as the physical database schema for an application. Clearly, the database architecture specialist for the type of database being used should be involved in defining and implementing the persistence store for his platform.

Benefits

Hopefully, by now, the value of a data architecture is becoming clear. Among other things, it provides:

• A mechanism for communication with all stakeholders in the business;
• A tool for impact analysis to determine the outcome of changes made;
• A strong mechanism for integration between heterogeneous systems;
• A mechanism for integration with external (partner/vendor) systems; and
• Governance over the way that data is stored and manipulated.

Once the data entities in an enterprise have been defined, the enterprise is much more able to adapt to changes in the business.

Where do you start?

Most data architecture initiatives fail very quickly because the sponsors fail to realize the following:

• A data architecture is a living model. It evolves as the business grows;
• Shutting 20 people away in a room for two years to develop a data architecture simply won't cut it. You cannot boil the ocean;
• The job is never done. The data architecture will continue growing permanently so any effort needs to be seen as a long-term program;
• Quantifying the benefit of data architecture is hard. It's easier to justify when you can look back and say "Here… we failed here and it cost us 20 million because we didn't have data architecture in place." 

Ideally you need to start with executive sponsorship from CxO-level people. That doesn't always happen, but often it is possible to start by using the approach I have highlighted here. Derive your business entities for a project, build these into an enterprise data model that can grow over time, implement a canonical model that will also grow as the business evolves, and be fastidious about making sure that you know what all the processes are that operate on the data. 

At the beginning of this article I talked about Data Strategy as opposed to data architecture. Ideally you want to attack the architecture as part of an overall strategy that addresses the issues that I raised above. The key thing that data architecture can help with, is better quality data and that is the lifeblood of modern business. If the data is correct, it is easy to extract virtually any intelligence that can be used to support business decisions, and that is the focus of information architecture.

My recent posts about integration with Microsoft Exchange have reminded me how important it is to keep the pragmatism of the application of technology to solve business problems very sharply in focus. During the process of building the Microsoft Exchange integration components, I have also been involved in a couple of projects that are very enterprise-focused during my day job. The combination of these projects made me step back and think about how we build software, and I realized that during the 20-some years that I have been in this industry, there are some things that I have learned and applied that have become the cornerstones of how I approach software. Later in this article, I am going to come back to this because it is key to the importance of Enterprise Architecture. 

Enterprise Architecture

As the term "Enterprise" suggests, enterprise architects are responsible for architecture that is enterprise-wide. That is, they are responsible for looking at the entire organization and understanding how it is structured to do business, identifying the areas where the process can be improved – often by the introduction of a technology solution – and then introducing the business process improvements and technology solutions that streamline the business.

Mature enterprise architecture actually goes even further than that. In a mature organization, enterprise architects work with the business leaders to define and enhance future business strategies, define processes to meet the strategies, and then implement those processes, using technology where appropriate.

What this means is that enterprise architecture is as much about understanding business as it is about technology.

Enterprise Architecture Disciplines

As with any discipline, enterprise architecture has numerous facets. The diagram at left shows all of the disciplines that are associated with enterprise architecture and provides a visual queue for how they relate to each other.

Disciplines in blue are all predominantly business-focused, whereas those in brown are more technology-focused. The lower down the stack you go, the more technology-focused the discipline is, whereas the higher up the stack you go, the more business-focused it is.

The oddball in the stack is Research and Innovation. This is a discipline that needs to span technology and business.

There are a few important things to bear in mind about these disciplines.

First, none of them lives in isolation. They all need each other to successfully deliver value to the enterprise.

Second, none of them is more important than the other. Just because Business Architecture is at the top of the stack does not mean that business architects are more important than database architects. Rather, the stack is intended to show how these disciplines relate to each other.

Third, a lack of focus on any one of these disciplines creates a gap that makes the stack incomplete. Inevitably, the organization will compensate for the lack of focus on an element in the stack by creating a compromised design. Nowhere is this more evident than in the Data Architecture discipline. If there is no focus on data architecture, an architecture will emerge organically, almost always generated from the schemas of component applications. The result is that integration suffers. 

Lastly, every discipline in the stack is directly dependent on the discipline above it. So information architecture requires a thorough definition of the business architecture in order to produce the business intelligence that is needed. Data architecture needs a clear information architecture to provide the data that is needed for the business intelligence to be derived, and so on.

An argument could be made that enterprise architecture governance is also a part of this stack. I am not opposed to that, but I really view governance as a management function that is embedded in each of the architectural disciplines. In some organizations, that governance does need to be broken out into a separate unit.

Let's take a look now at some details of what each of these disciplines does.

Research and Innovation

Research and innovation is really the oddball in the stack, because it spans the entire enterprise architecture stack, from business architecture all the way down to specialist architectural disciplines. in fact, it goes even lower than that, but that is beyond the scope of this discussion.

Research and innovation is about understanding what is going on in all industries that are related to the enterprise, understanding advancements that are being made that could enhance the business, and finding ways of exploiting those advancements to create competitive advantage. It also includes incubating new ideas that could allow an enterprise to leap-frog its competition.

In a future article, I will spend some time talking about how the research and innovation component of the Microsoft Exchange integration project has provided this kind of competitive advantage.

Business Architecture

It is hard to understate the importance of business architecture, and I think it is really interesting that in the last 10 years, IT has felt the backlash of driving businesses to make decisions based on technology, rather than sound business principles. Leading into Y2K, two massive technology drivers were forcing businesses to invest exorbitant sums of money in projects that were little more than experiments in technology – the Y2K "bug" and the advent of internet commerce. Much of that investment returned little to no value, and today IT organizations are being forced to focus their attention on business value. Business architecture is the architectural discipline that leads this initiative.

What this means is that business architects are really focused on understanding the business, not the technology. In a mature organization, business architects (and business analysts) work very closely with business units to understand their business processes. They are responsible for analyzing how the business runs, producing the definition of the business processes, defining new processes that align with the strategic goals of the enterprise, and defining the requirements from which all technical decisions are derived. A very important part of a business architect's role is to evaluate the existing processes to reduce redundancy, streamline the process, and automate as much of it as possible. Highly skilled business architects are also able to define the key performance indicators within a line of business so that it is possible to measure the success of the business.  

Information Architecture

Information architecture is the process of understanding what information the business needs, how to go about delivering it, and ensuring that nothing is lost. As with business architects, information architects have a strong business focus. They are normally lateral thinkers who are capable of envisioning ways to transform data into actionable business intelligence. Over and above the general reporting requirements that is part of the responsibility of information architecture, the discipline also involves understanding interactions with external stakeholders such as vendors, partners, and customers and producing an information strategy that monitors and enhances these interactions.

As with the business architecture discipline, information architecture is highly business-focused with its purpose being to provide decision makers the information they need to maximize business profitability.

Data Architecture

Probably the most neglected discipline in most companies is the data architecture discipline. Many people see data architecture as a function of information architecture, and it is probably best kept as part of the information architecture discipline, but it is really the first layer in the architecture disciplines that crystallizes business concepts into technology artifacts. What I mean by that is that data architecture is all about understanding the nature of the data in an organization, what operates on it, and how it is stored.

The data architecture discipline is absolutely crucial to integration because data architecture is the custodian of the enterprise data model that defines entities and their relationships across the enterprise. It therefore affects every other discipline around it, including application architecture, infrastructure architecture, solution architecture and integration.

The data architecture discipline also owns the responsibility of defining and maintaining the canonical model that is used in a service-oriented architecture to govern data transformations between services. 

My next post will include a detailed discussion of this discipline, especially as it relates to the Microsoft Exchange integration project.

Application Architecture

Application architecture is the process of understanding how applications operate on the data that is defined by the data architecture discipline. It is also responsible for understanding integration between applications so that they are able to properly communicate with each other and provide each other any necessary data.

Application architecture is also the custodian of the library of services that are required for a service-oriented architecture and they own the definition and governance of that library.

Infrastructure Architecture

Infrastructure architecture is the process of understanding the "plumbing" that is used to support a technical solution. So, for example, in the Microsoft Exchange integration case, Microsoft Exchange is an application and the OpenEdge CRM application is an application. These two applications need to communicate with each other. One (the OpenEdge CRM application) runs on a Unix server and Microsoft Exchange runs on a separate set of Windows servers. At a lower level, the OpenEdge CRM application has an AppServer associated with it and Microsoft Exchange uses a web server and has several server processes that are associated with it. With the roll out of the Microsoft Exchange integration solution, additional Java-based services such as Apache Tomcat and Apache ActiveMQ will be added to the mix.

Infrastructure architecture is about understanding these technical components, how they relate to and communicate with each other, and where the opportunities are for optimizing these infrastructure components.

As we move closer to cloud computing, this is a discipline that requires increasingly skilled people because virtualization, high-availability, network fabrics and all of those components factor into the decisions that are being made about hosting technology solutions.

The Divide between Business and Technology

Up until this point, all the disciplines that we have discussed have been focused on the business. Most of the questions that are being asked in those disciplines are related to how the architecture can be built to drive increased revenue and reduce operating costs. Many view this point as the division between what they call Enterprise and Technical architecture. In order to ensure maximum business benefit, though, these disciplines all need to work very closely with each other in a tightly cohesive team. I, therefore, view technical architecture as a facet of enterprise architecture and prefer to view technical architects as specialist architects who have extensive knowledge of their particular discipline.

Solution Architecture

Solution architecture is the discipline that most people think of when they talk about a Software Architect. A solution architect is able to take a specific problem, identify the appropriate way to solve the problem, and turn that into a technical solution that spans numerous specialist disciplines. The key point about this discipline, though, is that it is more technically oriented than business oriented. Successful solution architects are often viewed and technical gurus, capable of solving any technical problem.

Specialty Architecture

Specialty architects are extremely valuable resources as they understand their area of specialization inside-out. Generally, specialty architects have been in their field for several years and have solved significant business problems with the technology with which they are proficient. The above diagram references several time of specialty architects.

Integration architects have a thorough understanding of integration technologies like middleware, complex event processing, and business activity monitoring. In a service-oriented architecture, this discipline is responsible for overseeing the implementation of the canonical model and all related services. 

Application-specific specialty architects focus on products like SAP where there is a high level of skill required to understand the application as a whole. The other specialty architects are also highly skilled in their particular avenues and able to advise, mentor and lead significant development efforts based on their particular technology.

While it is likely that specialty architects are exceptional programmers, the discipline itself does not requires them to write code. 

Structured Systems Analysis and Design Methodology (SSADM)

What I find really interesting is that these disciplines have really only become clearly defined in the last few years. The reason is that 20 years ago, computer systems were nowhere near as complicated as they are now. There were far less of them and they were far more monolithic.

Nevertheless, 20 years ago I attended a course on a methodology called Structured Systems Analysis and Design Methodology (SSADM). In those days, systems analysts were expected to perform the business analysis and architecture function. So they would be responsible for documenting the business processes that the business needed.

From those business processes, a systems analyst would derive a set of data entities from the connecting lines between the processes. They would flesh out those entities by reviewing the paper forms or screenshots of existing applications to understand what the data elements were that made up the application. 

Those same analysts would then create a matrix (called an Entity-Event Matrix) that contained each process in the columns across the top of the matrix and each entity in the rows down the left side. They would then mark with a C, R, U, or D any cell that was affected by either a create, read, update or delete of the entity. From this, an entity life-cycle could be produced that demonstrated the life history of an entity. 

This matrix and the life-cycle were then used to build a specification of the processes that had to be built to operate on the data. Next, a set of JAD (Joint Application Design) sessions with the user would define the set of screens that would be built to enable the user to interact with the system and the reports that would be generated from the system.

Once the design is complete, the systems analyst and a group of programmers would go off and build the application.

As I said, this was a methodology that I was taught 20 years ago, and although much of it has been augmented by additional requirements imposed by additional complexity of modern computer systems, at the core, nothing has really changed that significantly. TOGAF and Zachman, two more modern enterprise architecture methodologies, emphasize the same basic principles of enterprise architecture. Of course, it is a much more iterative process now.

Conclusion

In many enterprises, architects are expected to be proficient in more than one of these disciplines. In large enterprises, although there is an expectation that enterprise architects are proficient in more than one discipline, often their focus is narrowed considerably by the amount of work to be done.

Nonetheless, I have found that my experience with SSADM taught me many of the things about enterprise architecture that I never knew I was going to value, and now, 20 years later, I am fortunate that I am able to cover the entire stack of enterprise architecture disciplines and many of the specialty disciplines, thanks to what I learned from SSADM.

Enterprise architecture is a relatively new buzzword that is increasingly being seen to provide significant business benefit and substantial return on investment. Look for these disciplines to grow significantly over the next 5 years as IT systems become increasingly complex. 

In the upcoming weeks, I will be building on this discussion by delving into each of these disciplines in more detail. First on my list is data architecture, as it is a discipline that is very near and dear to my heart.