Working without a data strategy is analogous to a company allowing each department and each person within each department to develop its own financial chart of accounts. This empowerment allows each person in the organization to choose his own numbering scheme. [...] Even to those of us who don't wear green eyed shades, the resulting chaos is obvious and easy to predict – (pg. 3, para 2).

What astounds me is how a risk that seems so blatantly obvious is completely missed by the majority of enterprises. In a recent study by Forrester Research, they found that 74% of over 400 companies surveyed view data strategy as critical or very important, but only 17% of them had a mature data strategy in place – (Topic Overview: Information Architecture by Gene Leganza, January 21, 2010, Forrester Research).

When you consider that most enterprises are outsourcing a substantial part of their core business systems, it is frightening that they do not have a strategy in place. The result is that each of their vendors defines their own view of the data and the enterprise loses control of what happens with their application infrastructure.

Scott Busse, in an article entitled Describing a Data Strategy to a Business Leader, called this uncontrolled, evolutionary data strategy a "waxy build-up" that leads to "higher costs, rigid processes, and a lack of insight into enterprise data" – (para 4). 

In this article we will briefly look at what Data Strategy is, and then focus on how data architectural integrity can be maintained in the Enterprise Architecture process.

The Scope of this Article

While the data architecture discipline owns the responsibility for the overall data strategy, I want to narrow data architecture for the purposes of this discussion. A formal, mature data strategy supports the enterprise by providing at least the following:

• Enterprise Data Model – The EDM is a model that defines all enterprise-level entities at a logical level, their relationships to each other, the life-cycle of each entity, what systems and services act upon it, and where the entity is used in the enterprise. It also defines the attributes of each entity and forms the basis for a common language for the enterprise.
• Canonical Model – The Canonical Model is a physical data model that is used by all applications and services to interact with each other throughout the enterprise. We'll talk more about the canonical model later.
• Metadata Management – Metadata is data that describes the entities and attributes in the Enterprise Data Model. So, for example, an account number may consist of 10 digits, of which the first 9 are a sequence number and the last digit is a check-digit. All of that information is metadata about the account number. This is a simple example, and in a large enterprise data model, there is a lot of metadata associated with the model.
• Data Quality – The GIGO principle (Garbage-In, Garbage-Out) is absolutely true about data and when strategic business decisions are being made based on invalid data, the results can be catastrophic. Ensuring that data integrity is maintained is a key part of any data strategy, and it has to be enforced in the data architecture as much as it is in the processes that govern the data.
• Data Governance – Managing and maintaining data from an operational point of view, ensuring that it is backed up, determining how long it is maintained in various stores, ensuring that the data stores perform well, and developing a high-availability and disaster-recovery policy all falls under the banner of data governance.
• Security and Compliance – Security and Compliance need to be closely aligned with the Data Governance component of a data strategy, but these functions are responsible for ensuring that legal compliance with things like SOX and PCI are taken care of. Security is responsible for ensuring data availability to the authorized channels while securing it from unauthorized ones. 

This is far more than I plan to focus on in this article. I want to talk about how you go about deriving the Enterprise Data Model components and the Canonical Model from the business process models that were created for the Exchange Integration Project that I blogged about a week or two ago.

The Business Process

The process at left deals with what happens when a sales person initiates a call with a prospect. The sales person initiates the call.

If the call does not succeed (i.e. the call fails, or the prospect is not available to take the call), the sales person documents the fact that the call was attempted in the system.

If the call goes through (i.e. the prospect is available for a conversation), a discussion is had and the sales person documents the conversation. Once the call is complete, the sales person documents the details of the call and determines the next steps to be taken, including any follow-up calls.

There is no question about the fact that I am way over-simplifying the business process in this discussion, but in order to keep this post as concise as possible, I need to do that. You can assume that some of the items that are noted as activities really need to be spelled out in more detail, and the business processes that are documented in the System swim-lane need a lot more information, but please cut me that slack… I need to keep this explanation short.

The Business Entities

One of the first things we can do with this model is define some of the business entities in this enterprise. In this process we have two entities who are really actors in the process:

• Prospect – The person who may be interested in goods and services from the enterprise; and
• Sales Person – The person responsible for pitching the sales of the goods and services to the prospect.

There are also several pieces of information that we care about that are flowing between these entities:

• Call – The fact that a call took place is important. We need to know the date and time of the call and how the call progressed;
• Discussion – The discussion that took place between the sales person and the prospect probably contains some very important information that would be useful to the enterprise if it were recorded; and
• Next Steps – The action that is to be taken as a result of the call could range from a follow-up call to the generation of a quote, or even an order.

So let's take these entities and model them to get an idea of what they look like.

 All of the entities that I references above are now part of this business entity model. We obviously have the prospect and sales person. We also have a call and a call next step. You'll note that the discussion got absorbed into the call as an element.

Now clearly, there is a lot more work that can be done to dig into each of these entities and refine them. Through an iterative cycle with the business, it is not hard to understand that a sales person is generally responsible for a prospect.

Moreover, a sales person is technically either an employee or contractor. That means that their information is available from other sources and in this context, they are simply performing a role in the business.

By the same token, a prospect may very well be an existing customer, in which case you want make sure that the information that you have for this prospect is stored for the customer. When the customer is a large organization, the prospect may, in fact, be a new department in the organization to which you have not sold before.

If a lead management system is present, the prospect may be an existing lead. So what is the difference between a prospect and a lead? Well a lead is a tip that you get about someone who may be interested in the product. A prospect is a lead for which the prospect-to-customer conversion process has already begun. 

A call is simply a way of interacting with a prospect. Other ways are by means of meetings, e-mails, letters, and other communication devices. Thus, the call is really an interaction.

Lastly, the call next step may be related to an appointment in the sales person's calendar, an order, or a quote.

So as you dig into each of these entities in more detail, it becomes clear that the model is significantly more complex and detailed than the business entity model we started defining above.

The point that is worth bearing in mind, though, is that these business entities were fairly easy to derive from the business process model that we discussed above. Although this process becomes more complex as the business processes are more complex, the same basic rules apply:

1. Each swim-lane is a major business entity;
2. Each line that crosses a swim-lane is a significant item of business data that needs to be tracked; and
3. By iteratively digging in to each of the business entities, it is fairly easy to identify numerous other affected entities.

Entity-Event Matrix

Once the business entities have been defined, it is important to understand the impact that each business process has on a business entity. To do this, I like to use a tool known as an Entity-Event Matrix. It is effectively a table with the list of entities across the top and the list of processes down the side. Each cell contains the actions that may be performed by that process on the entity. In the following table I have restricted my actions to create, read, update or delete, but in reality this becomes set of actions that specify the details of the changes made.

This matrix evolves as the model that it is associated with grows. Once the matrix has been completed for all business processes and aggregated together, any entity can be extracted from the matrix and a life-cycle for the entity can be produced.

At this point we have only discussed things from a business point of view. In a mature organization, the above deliverables are a function of the Business Architecture discipline (and I'm lumping Business Architecture and Business Analysis together).

The next step is to take these business entities and compare them against the Enterprise Data Model to determine whether there are any gaps.

The Enterprise Data Model

The terms "Enterprise Data Model," "Enterprise Conceptual Data Model," "Enterprise Logical Data Model," and "Canonical Model" are often bandied about in the industry somewhat interchangeably. So to make sure we are on the same page, I am going to define the Enterprise Data Model as follows:

An Enterprise Data Model is a logical model of the entities that exist across an enterprise, including their relationships, attributes and methods, as well as usage and value, and it is designed as a communication tool between all business, IT and external stakeholders.

There are several key things to bear in mind about the enterprise data model:

1. It is Enterprise-wide. That means that is the definitive guide to all entities that make the business work;
2. It is a logical model. That means that it does not have a physical manifestation, but it is used as the basis for defining the canonical model, which I will discuss a little later;
3. It is a communication device. That means that it is intended to help the business communicate with the business architects and all the other layers in IT. It effectively provides the common language that allows business units to properly communicate between themselves. It also serves as a reference point for external communication with partners, vendors, and outsourcing partners.
4. It is more than just an ER model. Beyond the entities and attributes that make it up, it also provides an impact analysis tool that allows the business to determine what systems are affecting different entities, where attributes are being realized, and what value data elements have to the business.

These are just four of the things that the Enterprise Data Model is; there are several others, but they are not germane to this discussion.

Once the business architects have done their due diligence, data architecture is engaged to validate the business entity model against the existing enterprise data model.

The model shown at left deliberately excludes attributes to simplify the diagram, but in this model, the data architect has taken the business entity model, mapped the entities and attributes to the enterprise entities and attributes, and identified any gaps in the enterprise data model. For these purposes, the calendar item entity is the only one that was missing.

In a mature organization, the enterprise data model is subject to a rigorous release cycle, much the same way as application software is.

Of course, the enterprise data model is not only affected by changes that are introduced by new business requirements. Often times, during the process of building components of the enterprise solution, additional entities and attributes are identified that should be included in the enterprise data model. Generally, this will trigger an architectural review of the enhancement as it is likely that other, more serious gaps exist. Nonetheless, it is very possible that additional entities and attributes can be introduced from sources outside business architecture. 

Once the enterprise data model changes are approved, focus shifts to the canonical model. 

The Canonical Model

Perhaps the most misunderstood, maligned concept in service-oriented architectures is the idea of a canonical model. Gregor Hohpe and Bobby Woolf spoke of the concept has a design pattern for enterprise integration in their book Enterprise Integration Patterns. While they were probably the first to document the pattern, there is nothing new about the idea. Many people have been using it for a very long time.

For example, about 12 years ago I was working on an EDI integration with a number of external parties. Each of the parties had a different standard and each of them needed data from one or more of the other parties. The simplest way to achieve that integration was to transform the data format from one party to a common format and then transform it to another format for the receiving party. This had nothing to do with SOA.

Actually, back as far as 1991, I was on a project that involved a conversion from several systems into a new system. Each of the old systems had data that contributed to the new system and all of the contributed data needed to be written in a single transaction to ensure referential integrity. The conclusion was to create a common model that could be addressed by all of the interested systems, and then transform data into that common model before moving it to the new system.

A canonical model is therefore a layer above all applications to which each application is able to transform their data to integrate with other systems.

In the context of our example, the canonical model is responsible for providing the transformation layer between the CRM application, the LDAP Server and the Microsoft Exchange Server. To create a calendar item in Microsoft Exchange Server, we need to draw data from the CRM application and the LDAP Server and then send it across to Microsoft Exchange in the correct format.

The CRM application has the Call Next Step data in a certain format. The LDAP server has directory information about the organizer and the attendees for the calendar item that needs to be created in Microsoft Exchange Server.

By creating a Calendar Item in the canonical model that conforms to a certain standard format, we can extract data from CRM and LDAP, transform it and store it in a canonical Calendar Item, ship the calendar item to the Microsoft Exchange Server, and transform it into the Exchange Web Service Item form that does not mean anything to anyone other than Exchange. Other systems that wish to make use of the Calendar Item can also perform their own transformations to their native formats.

More than just data, the canonical model also contains the methods that operate on canonical model objects. Moreover, the canonical model is often implemented as a relational model, an object-relational model, and an XML model. 

Persistence Models

Up until now, we have not dealt with persistence models at all – that is, models that are created to persist the data to a physical data store. As the canonical model effectively defines the entities and attributes for the enterprise, it can often be easily adapted for use as the physical database schema for an application. Clearly, the database architecture specialist for the type of database being used should be involved in defining and implementing the persistence store for his platform.

Benefits

Hopefully, by now, the value of a data architecture is becoming clear. Among other things, it provides:

• A mechanism for communication with all stakeholders in the business;
• A tool for impact analysis to determine the outcome of changes made;
• A strong mechanism for integration between heterogeneous systems;
• A mechanism for integration with external (partner/vendor) systems; and
• Governance over the way that data is stored and manipulated.

Once the data entities in an enterprise have been defined, the enterprise is much more able to adapt to changes in the business.

Where do you start?

Most data architecture initiatives fail very quickly because the sponsors fail to realize the following:

• A data architecture is a living model. It evolves as the business grows;
• Shutting 20 people away in a room for two years to develop a data architecture simply won't cut it. You cannot boil the ocean;
• The job is never done. The data architecture will continue growing permanently so any effort needs to be seen as a long-term program;
• Quantifying the benefit of data architecture is hard. It's easier to justify when you can look back and say "Here… we failed here and it cost us 20 million because we didn't have data architecture in place." 

Ideally you need to start with executive sponsorship from CxO-level people. That doesn't always happen, but often it is possible to start by using the approach I have highlighted here. Derive your business entities for a project, build these into an enterprise data model that can grow over time, implement a canonical model that will also grow as the business evolves, and be fastidious about making sure that you know what all the processes are that operate on the data. 

At the beginning of this article I talked about Data Strategy as opposed to data architecture. Ideally you want to attack the architecture as part of an overall strategy that addresses the issues that I raised above. The key thing that data architecture can help with, is better quality data and that is the lifeblood of modern business. If the data is correct, it is easy to extract virtually any intelligence that can be used to support business decisions, and that is the focus of information architecture.

In a follow-up post, Ms Manes pointed out that  "SOA is a bad word. And yes, doing SOA is a good thing. (And we need to keep doing it!) But no, we do not need to come up with a new name. Emphatically not."

This morning my RSS feeds had an update from Ms Manes’ blog again. She points to an article by Dan Woods of Forbes.com and says of his article that she is "pleased to see that Dan read beyond the first paragraph, and he understands the core message of my post (i.e., ‘SOA has been disappointing and that services should be a key focus’)" and then goes on later in her post to say:

"Dan recommends an incremental approach to SOA: Just build services as you need them on a project-by-project basis, and at some point in the future you can go back and consolidate the services you’ve built and somehow derive some architectural consistency. Unfortunately, this strategy invariably leads to Just a Bunch of Web Services (JABOWS) because the future consolidation step almost never happens. The result is too many services, too many moving parts, and too many brittle connections. Systems wind up being more expensive and more fragile than ever before."

I wonder if Ms Manes realizes that she advocated exactly the same thing that Mr Woods advocated and then went on to slam the idea? In her original post she argued that the focus should shift from SOA to building services, and yet here she is arguing that just building services will result in fragile, expensive systems. Ms Manes is just flat-out contradicting herself.

Without the overall service-oriented architecture you will not get a cohesive system. So recommending building services without an overall service-oriented architecture is a bad idea… as Ms Manes suggests.

At the same time, Ms Manes does not like the term "SOA" because of the negative connotation it has garnered thanks to a lack of understanding that business need to make a significant commitment to change and there are unscrupulous individuals trying to profit from a technology buzzword. For those reasons, Ms Manes believes SOA is dead. Ms Manes, however, does not believe we should get rid of the word "SOA".

In her latest post Ms Manes makes the statement that "One of the reasons SOA ‘died’…" Just stop right there!!!

Whether you like the term SOA or not, service-oriented architecture is a critical part of Enterprise 2.0 and requires a significant investment in both the architecture and development of services. Service-orientation has to drive the way that development teams build applications today and into the future.I hate to tell you this, Ms Manes, but SOA is not dead. There are numerous organizations who are still working extremely hard on a service-oriented architecture and many more are seeing the importance of it more and more with things like mashups and social networking becoming tightly integrated into the business.

I don’t disagree that SOA requires governance, management, leadership, skills, practices and a commitment to change. But remember that change is hard and takes time. In fact, it takes more time as the organization gets larger. A weak economy has caused businesses to refocus priorities and in many cases this has meant cutting projects that are not directly contributing to the bottom line. Companies are not willing to shell out several hundred thousand on projects for which the ROI may only be seen in 5 to 10 years. For some companies, that is the amount of change that SOA requires and simply buying an ESB is not going to resolve the issues.

A service-oriented architecture does not come from the presence of an ESB. It is definitely not characterized by the availability of a bunch of services in an organization. SOA is a paradigm shift like event-driven programming was a paradigm shift and object-orientation was a paradigm shift. All of them came with their share of snake oil salesmen too and nowadays those paradigms are standard in business.

SOA is alive and well… and growing.

Now you may be sitting there reading this thinking "OK… that was back in January. Why are you still writing about it now?"

Well since then the blogosphere has been on fire with people chiming into this debate asserting that "SOA is too complicated" or "ESBs have failed" or "businesses cannot afford SOA" or some other statement along those lines, implying (sometimes directly stating) that SOA was a bad idea and that it deserves to die.

Even people like Brent Foody at Progress Software agreed with Ms Manes when he wrote about this in his post. As Progress is normally pretty good at ensuring that its own corporate blogs meet the business goals of the company and given they own Sonic Software, vendor of the Sonic ESB, it is surprising that they allowed a posting prematurely acknowledging the demise of an architecture that they promote. Add to that the fact that Mr Foody's comments contradict David Chappell's own thoughts on the matter and you can see why I am surprised. Remember, David was pretty much the architect of SonicMQ and the brains behind SonicESB.

But even those were posted way back in January. As late as last week, though (July 23rd), there was still fall-out from this post. ZapThink hosted a ZapForum entitled "Night of the Living SOA Dead" where people would get the chance to argue the whole thing out. So this story has had legs for nearly 7 months.  

When I read Ms Manes' post, I felt like the person standing on the sidewalk in a movie. The camera is focused on me and someone (SOA) steps out into the street. From the side of the screen, off-camera, a bus comes careening past and the person that stepped into the street is hit and taken completely out of camera and I'm left standing with my jaw on my knees.

Of course, if you carefully read Ms Manes' post, you have to agree with at least some of what she is saying:

• The economy cannot support SOA projects that do not provide good ROI in a fairly short period of time;
• There is nothing to be gained by SOA for the sake of SOA;
• SOA is not the hammer for every nail;
• SOA is best viewed as part of an application modernization strategy for breaking up the monolith; and
• SOA is not the silver bullet that will fix everything your organization.

Ms Manes' argument that the state of the economy would lead to the demise of SOA leaves me more than a little perplexed. Sure. Companies will cancel SOA projects that do not have significant ROI in the short or medium term in order to focus financial resources where they are most needed, but that does not spell the demise of SOA. Rather, it does what needed to be done in the first place – focuses business priorities on the highest ROI first.

Many vendors have bandied the SOA acronym around as the mechanism for selling their wares - from ESBs to high-cost consulting gigs; from application servers to expensive monitoring tools, all in the name of a service-oriented architecture. There is no doubt that a number of companies have excellent products that are intended to assist in building out your service-oriented solution, but just because you have an ESB does not mean that you have a service-oriented architecture. Neither is it true that because you have deployed several WebServices you now have a service-oriented architecture.

Ms Manes' arguments are actually not new. In fact, many of her comments very much echo the sentiments of Thomas Erl, author of "Service-Oriented Architecture – Concepts, Technology and Design".

Back in July 2005, Mr Erl contended that "the rise of the false SOA [one characterized by the idea that a set of deployed WebServices represents a service-oriented architecture] has distorted the vision of the ideal SOA. Not only is the false SOA divergent from the 'true path of service-orientation,' it reinforces SOA anti-patterns by extending and further entrenching the traditional distributed computing model to which SOA offers an alternative." Essentially, Mr Erl makes the point that unless your implementation of SOA follows a well architected strategy, you are doing nothing more than creating a more complex distributed computing system.

With the amount of money that has been thrown at SOA, business has a right to expect that the problem that SOA was intended to resolve has been resolved. The issue is that there have been too many snake oil salesmen delivering products to a site and then leaving the organization to find its own way. Sometimes, the 'snake oil salesman' actually sold a very good technology into the site with an offer of professional services to establish a competency center, but the customer failed to understand the importance of a supervised architectural change to the environment. As Mr Erl observes, "A real SOA requires real change, real foresight, and real commitment. Most of all, though, it requires guidance." – (Italics my own).

This brings me to my disagreement with Ms Manes' point of view. Ms Manes states "Successful SOA (i.e., application re-architecture) requires disruption to the status quo. SOA is not simply a matter of deploying new technology and building service interfaces to existing applications; it requires redesign of the application portfolio. And it requires a massive shift in the way IT operates." She follows this up by saying "The latest shiny new technology will not make things better. Incremental integration projects will not lead to significantly reduced costs and increased agility. If you want spectacular gains, then you need to make a spectacular commitment to change." In these statements Ms Manes' is asserting that business somehow expects an overnight change and that a successful service-oriented architecture can only come from turfing your existing architecture and replacing it with a new SOA-based solution. Not only is that not true, but it is also hopelessly unrealistic.

Yes, SOA requires a different way of thinking and paradigm shifts are not going to come overnight. And yes, deploying a new technology and building service interfaces does not constitute a service-oriented architecture, but it is an extremely important transitional step in moving a commercial environment to a more service-oriented architecture.

Ms Manes ends her post by saying "And that’s where we need to concentrate from this point forward: Services." In so doing, Ms Manes is opening us up to the same "false SOA" that got us into this mess in the first place.

Ultimately, a thorough service-oriented architecture has proper governance and standards in place across all deployed services to ensure the abstraction of the underlying components. If that governance and standardization is not in place, services are simply distributed components.

So, no. SOA is not dead. A service-oriented architecture is something that businesses definitely need to aspire to achieve. It's probably safe to say, though, that the 'snake oil' SOA days are over.