In the last post I had given a primer on service oriented data warehousing. I would like to dwelve deeper into some of the topics that SOA and DW holds in parlance.

One of the most important topics is “DATA”, where DW community had learned valuable lessons over years of experience in data warehouse implementations. The same is case with SOA, where pioneers of SOA have learned valuable lessons on data quality, data integration, and governance.

Any SOA initiatives will fail:

• When they sit on a platform of poor quality data,
• Attempt to share data that is not integrated across the enterprise,
• Implemented with insufficient thought given to security, compliance, and change management.

It strikes to me about “abstract services” in the data warehouse world, which are commonly recognized, simply stated, and independent of specific data sources, business processes, and BI deployments. I guess we can do that, let’s see how this works.

Consider the relationship between the dimension manager and the fact provider.

A dimension manager is the centralized resource for defining and publishing a conformed dimension to the rest of the enterprise. A master data management (MDM) resource is an ideal candidate for dimension manager. Enterprises where MDM is not instititionalized, the data warehouse team is a kind of “downstream MDM” function that gathers in-compatible descriptions of an entity such as customer and publishes the cleaned, conformed, as well as de-duplicated dimension to the rest of the data warehouse community.

The subscribers to this dimension are almost always owners of fact tables who want to attach this high quality conformed dimension to their fact tables so that BI tools around the enterprise can perform drill across reports on the conformed contents of the dimension.

There are two basic services that every dimension manager publisher needs to provide to the fact table subscribers.

Fetch means that fact table provider pulls information from the
dimension manager.

Alert means that the dimension manager pushes
information to the fact providers.

Let me elaborate;

• Fetch specific dimension member (we assume in this and the following steps that thedimension record has a surrogate primary key, a dimension version number, and that the information transmitted is consistent with the security and privacy privileges of the requester).
• Fetch all dimension members.
• Fetch dimension members changed since specific date-time with designated SCD Types 1, 2, and 3.
• Fetch natural key to surrogate key correspondence table unique to fact table provider.
• Alert providers of new dimension release. (A major dimension release requires providers to update their dimensions because Type 1 or Type 3 changes have been made to selected attributes).
• Alert providers to late arriving dimension members. (Requires fact table provider tooverwrite selected foreign keys in fact tables).
  
  These services are generic to all integrated data warehouses.
  
  In a dimensionally modeled data warehouse, we can describe the administrative processing steps with great specificity, without regard to the underlying subject matter of the fact or dimension tables.

That is why, in SOA parlance, dimensional modeling provide,s well-defined reference architecture on which to base these services. These services seem pretty defensible as meeting SOA design requirements.

SODW Success is Enterprise Success

Mantra for Successful SODW

1. A focus on the nouns and the verbs of your organization to provide guidance in decision making when contemplating new projects. Does the proposed project fit within and/or adhere to our organizational definitions and requirements of nouns (data) and verbs (function)? This new system must integrate with the rest of the enterprise.

2. Data governance programs and policies to define clear ownership, purpose, and use of data in the organization.

3. Function governance programs and policies to define clear ownership, purpose and use of shared functions and services in the organization. As data management professionals, we all know the importance of data governance, so we are all prepared for making sure our functional architecture has the same discipline and rigor.

4. A Master Data Management (MDM) strategy that supports the integration of and use of the shared data in the organization. MDM, in my mind, is one of the key components to any effective service oriented architecture (SOA), and often gets overlooked. You cannot effectively share services if you do not effectively share the data that is the backbone of your organization.

5. Data integration and data access services are the core services that allow us to integrate data into our MDM solution and warehouse, as well as provide common access to that data. This is where we make data integration lighter and provide the ability to access analytic data more dynamically with other, non-analytic processing. This is also where we can integrate the rest of our organization’s data, the semi and un-structured data.

6. Data quality and standardization services to make sure that the shared data being integrated into the MDM solutions are all applying the same rules. In fact, it would be best to have these services pushed as close as possible to where the data enters the organization, whether it is typed in on a web page or comes in as an XML transaction from a 3rd party. Many data warehouse solutions do the data quality and standardization; but if I can leverage these as services and put them at the point of entry then we will have better data across the organization, making integration easier and more cost effective.

7. A technology platform that facilitates the deployment and orchestration of data services as part of every day business processes. Something like “closing the loop” in data warehousing. This is where it becomes possible to leverage data and functionality that was exclusive to the data warehouse and business intelligence applications and make them part and parcel of everyday business processing.

To be successful in deploying a SODW, your organization must realize the increased interdependency with the rest of the applications in your organization, which is a good thing. Previously we moaned about the hassles with poor data governance upstream from the data warehouse. Well, now we also have to worry about having a consistent, interoperable function architecture as well. Again, in my mind this is a good thing. It is an opportunity, initially borne out of the need to integrate data, and now function, that will push the data warehouse to the center of the organization.

To conclude the data warehouse is no longer a proxy for the enterprise data architecture, but a driver for better overall enterprise architecture which is sharing, reusing, standardizing, increasing efficiency… Sounds an awful lot like saving money to me.

Service Oreinted Data Warehousing - SODW

When I sat down to write this blog on service oriented data warehousing (SODW), I had to reflect on the work and struggles I have had in data warehousing and data integration, the people I had learned from, and some of the failures trying to push these concepts before they were technically feasible. From those reflections I came to realize that

1) a data warehouse is a proxy for good enterprise data architecture, and

2) the process for which these things are being built makes them very difficult to change as the needs of the organization changes. Not to mention that most organizations have separate groups that do operational integration and data warehouse integration (ETL).

Data warehouses, to some extent, have become part of the problem they intended to correct: another system that demands resources for its care and feeding, yet doesn’t really integrate well with the other systems.

How can we build solutions that allow us to integrate data for both operational and analytic purposes?

If we can integrate data more easily, would that not facilitate operational analytics and active data warehousing?

How do we integrate and tap into all that semi-structured and unstructured data?

To satisfy these questions, to really close the loop and make the warehouse part of the organizations daily activity, is to rethink how we integrate data and expose that data to the organization.

I don’t want to state the obvious here but many data warehousing and business intelligence projects are not as successful as they could be because they lack a sufficient enterprise foundation. There are many things that must be in place such as data that is standardized and of a high quality, consistent definitions of information and function, and ownership of data and function.There is an important point I need to make here: the need for a solid foundation doesn’t change just because we are throwing around the “service oriented” tag. Let’s not confuse the need for flexibility and agility with quick and dirty. SODW still requires good enterprise data architecture practices that are aligned with how the data is used. Perhaps even more discipline and rigor are required than we have seen in the past. In fact, a successful SODW program will help drive improvement by focusing on doing things well once and reusing them, instead of doing the same things many times quickly. In the long run, reusability is more cost effective and more agile.

So what do we need to for successful SODW? Really, we need the same things across the enterprise to make things truly interoperable and reusable. These things are not only keys for successful SODW but, in my opinion, successful enterprise applications. I will explain them in my next post