Where is Real SOA

In my last post I was clear on what is happening to SaaS BI. Since then we had seen many changes in the SaaS area. One thing is clear. All these BI vendors out there are battling to survive. While traditional on premise vendors are announcing cloud offerings and niche players are still determining right technology and business model that can make them sustainable in the long run.

Is world business ready for such offerings. Its compelling in theory and logic. But logic will not apply always in real world. No sense sometimes is common sense.

I was working on SOA project for one of my client who had spent millions of money on SAP implementation. He has simple question. Can he get process analytics out of the processes that run in SAP. I jumped the bandwagon. I gave huge presentation to them on what SAP offers as Process Integration (PI) and CE (Composite Environment). What is Business Process Management offering from SAP.

How we can configure and create Enterprise Services in SAP in Enterprise Service Repository. It all looked well and nice. I got POC.

I started POC with my SAP consultant who created simple leave application and showed them how it can be done. Super, accolades all across.

I was asked to work on Procure to Pay process as project.

That's when reality checked in.

What we found was that SAP that's been implemented has got many exceptions embedded and its not the way SAP recommends. There are many process gaps which manifested in core SAP implementation. Its so difficult for me to enable the services because there are lot of ifs and buts.

Even the CIO was surprised rather shocked to see this. He asked me to configure the BPM as per SAP and then do GAP analysis. I am still working on it.

Point here is simple. technology out there enables us to configure and consume data and functionality through web services. All we need to do is do it right.

I wait for day where I can consume data visualization as service and add my data to it and consume it. Is it not possible in simple way, yes it is.

SaaS BI - Crossroads

We have to take serious look at SaaS BI. We see them make lot of noise and fall apart when it comes to deliver the promises they make. Some times they even lack basic direction.

One thing is for sure. SaaS BI needs serious transformation. In their strategies, technologies, operations and more importantly in the way they manage their finances.

There is always notion that SaaS BI vendors have created that they bring in low cost advantage and rapid deployment. By playing on cost and speed they seem to have lost some potential innovative approaches they can bring to the table. Not that every one are doing the same but most of them still talk about cost and speed. This approach has been commoditized. I guess its time for SaaS BI vendors to innvovate in their offerings. Bring in new dimension with respect to value add. Not just COST & SPEED.

Lets ask one basic question?

Why do traditional datawarehouse implementations fail?

All of us can write a book on the reasons. If we implement SaaS BI with same traditional BI technologies or mindset will they succeed.

Nope.

We are in world where users want to consume information in most simple manner. Amazon, Twitter, Facebook have changed world perception of IT systems. Most of the vendors still lock them in Traditional BI mindset. That needs to change and when it changes consumers will love it and they will flock to it.

off late we see lot of jargon predictive analytics, enterprise mashups, columnar data, in-memory data etc etc.

Lets see some of these in my next post.

MI and BI

I met senior official of data management group of Pharmaceutical Company as part of my consulting engagement. He came up with requirement called consumer analytics. Different pharmacy chains their products in market. He would like to get POS data from these chains, cleanse it, analyze it and mine the data to further insights. As we know there is no customer in this kind of situation, its traditional business intelligence coupled with market intelligence seems to be solution.

I always feel that very soon there seem to be convergence with BI and MI. By definition BI is with in walls of business where as MI is out side information.



If BI can be integrated with MI and analyzed it would solve lot of problems for businesses. This is more of case with situation I am mentioning above. Technically I am looking for SOA enabling external MI to be consumed and integrated with internal BI. There are lot of databases and market information available out side such as IMS for Pharmaceutical sector. I see most of these are still desktop applications that needs to be downloaded and installed. If these can be consumed as web services then accessing MI would be easier for companies.

As I work through my consulting engagement through available resources, I look forward to have a day where key MI tools deliver data as web services.

Specialty BI

As I work with CoE for SaaS and Cloud Computing at Bodhtree Consulting Ltd, I work with several starts ups which are mostly cloud based. There is one company which I feel is redifining the market in its area of operations.

And the company name is Analytix On Demand (AOD), LA based BI vendor.

AOD started as SaaS.

Delivered SaaS based BI services to its customers.

They have paying customers, good to hear that.

Story do not end there.

They strengthened the platform and started OEM deals.

They created industry specific verticals for clients with same base platform.

Is it not against cloud basics??????

Any technology guy would would yes.

But see the upside, they created platform and delivers it with successful business models.

This I guess is star strategy and I would like to categorize them as SPECIALTY BI vendors.

Vendors who are felxible to customer needs and understands market situation.

AOD with out hanging to technological fads is creating value in the marketplace.

They are strengthning platform and creating additional deployment and delivery models right from SaaS / OEM / Industry Specific Solutions.

And most of these are on cloud.

So I feel in these market conditions companies such as AOD who quickly adapt and add value to customers will sustain in longterm.

Enterprise Search in BI & SOA

Let me start this post by exciting you through what one could achieve through enterprise search in Business Intelligence. Think of following scenarios:-

• You can do a Google equivalent search on all published reports and analytics lying in your data publishing servers and BI engines.
• You can do a search on all the meta-data including hierarchies, data models, report structures, views etc.
• You can find out on what all reports and views, a given data is lying (best way to find inconsistencies).
• What if you can have a common interface of search, by which you can search the unstructured data and Business Intelligence data. For example, you may like to find out the technical specs of the products which are least selling in last 3 months. Etc..

Here are the natural benefits of Enterprise search in Business Intelligence:

• People can home into the information they are seeking without specifying the specific report.
• Enterprise search will definitely reduce the query load on the Business Intelligence systems, as people will find many of their queries being answered from existing reports, instead of firing a new query.
• Less training- People don't have to learn to make a query or understand the structure of reports repository. They simply need to understand on how to place a good search string.
• Combining Business Intelligence data with unstructured data completes the picture for the users.

Now let's see why the Enterprise Search in its nascent stage

• Market still has to wake-up to its potential, and Vendors have yet not offered effective, widely acceptable and convenient offerings.
• The enterprise search and Business Intelligence players are different specializations, and they still have to actively engage with each other.
• Frantic consolidation phase, leading to an eye off the innovation, with more focus on integration of the acquisitions.

The barriers to growth of enterprise search:

The enterprise search is more of a business and modeling challenge than a technology challenge. For an enterprise search to be effective, we need to have the following enablers, which will take time to achieve value add.

• A robust and well-defined meta-data model, and the user search needs to be guided by the meta-data model. Just imagine, if there are 100 different reports having hundred different terms of revenue. A user will see "Net Revenue" figures as being different across multiple reports, because inherent definition of "Net Revenue" may be different in these reports. Building a robust meta-data repository could be a big issue, as most of the organization struggle with that.
• Guided discovery of search: The Business Intelligence search will become effective, if there is a guided search mechanism in the search capabilities. This will mean that a person when he gets 100 reports as the search results, he should be able to provide intelligence clues to home onto the right report which will provide the data. This is different from adding some additional keywords in your search bar.
• Published reports will most-probably need to have "search tags" to ensure a faster and effective search. This will need technical changes in the products and also a more informed super-users.

Lets explore different ways including of using SOA to address the above barriers in my next post.

In the last post I had given a primer on service oriented data warehousing. I would like to dwelve deeper into some of the topics that SOA and DW holds in parlance.

One of the most important topics is “DATA”, where DW community had learned valuable lessons over years of experience in data warehouse implementations. The same is case with SOA, where pioneers of SOA have learned valuable lessons on data quality, data integration, and governance.

Any SOA initiatives will fail:

• When they sit on a platform of poor quality data,
• Attempt to share data that is not integrated across the enterprise,
• Implemented with insufficient thought given to security, compliance, and change management.

It strikes to me about “abstract services” in the data warehouse world, which are commonly recognized, simply stated, and independent of specific data sources, business processes, and BI deployments. I guess we can do that, let’s see how this works.

Consider the relationship between the dimension manager and the fact provider.

A dimension manager is the centralized resource for defining and publishing a conformed dimension to the rest of the enterprise. A master data management (MDM) resource is an ideal candidate for dimension manager. Enterprises where MDM is not instititionalized, the data warehouse team is a kind of “downstream MDM” function that gathers in-compatible descriptions of an entity such as customer and publishes the cleaned, conformed, as well as de-duplicated dimension to the rest of the data warehouse community.

The subscribers to this dimension are almost always owners of fact tables who want to attach this high quality conformed dimension to their fact tables so that BI tools around the enterprise can perform drill across reports on the conformed contents of the dimension.

There are two basic services that every dimension manager publisher needs to provide to the fact table subscribers.

Fetch means that fact table provider pulls information from the
dimension manager.

Alert means that the dimension manager pushes
information to the fact providers.

Let me elaborate;

• Fetch specific dimension member (we assume in this and the following steps that thedimension record has a surrogate primary key, a dimension version number, and that the information transmitted is consistent with the security and privacy privileges of the requester).
• Fetch all dimension members.
• Fetch dimension members changed since specific date-time with designated SCD Types 1, 2, and 3.
• Fetch natural key to surrogate key correspondence table unique to fact table provider.
• Alert providers of new dimension release. (A major dimension release requires providers to update their dimensions because Type 1 or Type 3 changes have been made to selected attributes).
• Alert providers to late arriving dimension members. (Requires fact table provider tooverwrite selected foreign keys in fact tables).
  
  These services are generic to all integrated data warehouses.
  
  In a dimensionally modeled data warehouse, we can describe the administrative processing steps with great specificity, without regard to the underlying subject matter of the fact or dimension tables.

That is why, in SOA parlance, dimensional modeling provide,s well-defined reference architecture on which to base these services. These services seem pretty defensible as meeting SOA design requirements.

SODW Success is Enterprise Success

Mantra for Successful SODW

1. A focus on the nouns and the verbs of your organization to provide guidance in decision making when contemplating new projects. Does the proposed project fit within and/or adhere to our organizational definitions and requirements of nouns (data) and verbs (function)? This new system must integrate with the rest of the enterprise.

2. Data governance programs and policies to define clear ownership, purpose, and use of data in the organization.

3. Function governance programs and policies to define clear ownership, purpose and use of shared functions and services in the organization. As data management professionals, we all know the importance of data governance, so we are all prepared for making sure our functional architecture has the same discipline and rigor.

4. A Master Data Management (MDM) strategy that supports the integration of and use of the shared data in the organization. MDM, in my mind, is one of the key components to any effective service oriented architecture (SOA), and often gets overlooked. You cannot effectively share services if you do not effectively share the data that is the backbone of your organization.

5. Data integration and data access services are the core services that allow us to integrate data into our MDM solution and warehouse, as well as provide common access to that data. This is where we make data integration lighter and provide the ability to access analytic data more dynamically with other, non-analytic processing. This is also where we can integrate the rest of our organization’s data, the semi and un-structured data.

6. Data quality and standardization services to make sure that the shared data being integrated into the MDM solutions are all applying the same rules. In fact, it would be best to have these services pushed as close as possible to where the data enters the organization, whether it is typed in on a web page or comes in as an XML transaction from a 3rd party. Many data warehouse solutions do the data quality and standardization; but if I can leverage these as services and put them at the point of entry then we will have better data across the organization, making integration easier and more cost effective.

7. A technology platform that facilitates the deployment and orchestration of data services as part of every day business processes. Something like “closing the loop” in data warehousing. This is where it becomes possible to leverage data and functionality that was exclusive to the data warehouse and business intelligence applications and make them part and parcel of everyday business processing.

To be successful in deploying a SODW, your organization must realize the increased interdependency with the rest of the applications in your organization, which is a good thing. Previously we moaned about the hassles with poor data governance upstream from the data warehouse. Well, now we also have to worry about having a consistent, interoperable function architecture as well. Again, in my mind this is a good thing. It is an opportunity, initially borne out of the need to integrate data, and now function, that will push the data warehouse to the center of the organization.

To conclude the data warehouse is no longer a proxy for the enterprise data architecture, but a driver for better overall enterprise architecture which is sharing, reusing, standardizing, increasing efficiency… Sounds an awful lot like saving money to me.