SaaS BI - Crossroads

We have to take serious look at SaaS BI. We see them make lot of noise and fall apart when it comes to deliver the promises they make. Some times they even lack basic direction.

One thing is for sure. SaaS BI needs serious transformation. In their strategies, technologies, operations and more importantly in the way they manage their finances.

There is always notion that SaaS BI vendors have created that they bring in low cost advantage and rapid deployment. By playing on cost and speed they seem to have lost some potential innovative approaches they can bring to the table. Not that every one are doing the same but most of them still talk about cost and speed. This approach has been commoditized. I guess its time for SaaS BI vendors to innvovate in their offerings. Bring in new dimension with respect to value add. Not just COST & SPEED.

Lets ask one basic question?

Why do traditional datawarehouse implementations fail?

All of us can write a book on the reasons. If we implement SaaS BI with same traditional BI technologies or mindset will they succeed.

Nope.

We are in world where users want to consume information in most simple manner. Amazon, Twitter, Facebook have changed world perception of IT systems. Most of the vendors still lock them in Traditional BI mindset. That needs to change and when it changes consumers will love it and they will flock to it.

off late we see lot of jargon predictive analytics, enterprise mashups, columnar data, in-memory data etc etc.

Lets see some of these in my next post.

MI and BI

I met senior official of data management group of Pharmaceutical Company as part of my consulting engagement. He came up with requirement called consumer analytics. Different pharmacy chains their products in market. He would like to get POS data from these chains, cleanse it, analyze it and mine the data to further insights. As we know there is no customer in this kind of situation, its traditional business intelligence coupled with market intelligence seems to be solution.

I always feel that very soon there seem to be convergence with BI and MI. By definition BI is with in walls of business where as MI is out side information.



If BI can be integrated with MI and analyzed it would solve lot of problems for businesses. This is more of case with situation I am mentioning above. Technically I am looking for SOA enabling external MI to be consumed and integrated with internal BI. There are lot of databases and market information available out side such as IMS for Pharmaceutical sector. I see most of these are still desktop applications that needs to be downloaded and installed. If these can be consumed as web services then accessing MI would be easier for companies.

As I work through my consulting engagement through available resources, I look forward to have a day where key MI tools deliver data as web services.

Specialty BI

As I work with CoE for SaaS and Cloud Computing at Bodhtree Consulting Ltd, I work with several starts ups which are mostly cloud based. There is one company which I feel is redifining the market in its area of operations.

And the company name is Analytix On Demand (AOD), LA based BI vendor.

AOD started as SaaS.

Delivered SaaS based BI services to its customers.

They have paying customers, good to hear that.

Story do not end there.

They strengthened the platform and started OEM deals.

They created industry specific verticals for clients with same base platform.

Is it not against cloud basics??????

Any technology guy would would yes.

But see the upside, they created platform and delivers it with successful business models.

This I guess is star strategy and I would like to categorize them as SPECIALTY BI vendors.

Vendors who are felxible to customer needs and understands market situation.

AOD with out hanging to technological fads is creating value in the marketplace.

They are strengthning platform and creating additional deployment and delivery models right from SaaS / OEM / Industry Specific Solutions.

And most of these are on cloud.

So I feel in these market conditions companies such as AOD who quickly adapt and add value to customers will sustain in longterm.

Enterprise Search in BI & SOA

Let me start this post by exciting you through what one could achieve through enterprise search in Business Intelligence. Think of following scenarios:-

• You can do a Google equivalent search on all published reports and analytics lying in your data publishing servers and BI engines.
• You can do a search on all the meta-data including hierarchies, data models, report structures, views etc.
• You can find out on what all reports and views, a given data is lying (best way to find inconsistencies).
• What if you can have a common interface of search, by which you can search the unstructured data and Business Intelligence data. For example, you may like to find out the technical specs of the products which are least selling in last 3 months. Etc..

Here are the natural benefits of Enterprise search in Business Intelligence:

• People can home into the information they are seeking without specifying the specific report.
• Enterprise search will definitely reduce the query load on the Business Intelligence systems, as people will find many of their queries being answered from existing reports, instead of firing a new query.
• Less training- People don't have to learn to make a query or understand the structure of reports repository. They simply need to understand on how to place a good search string.
• Combining Business Intelligence data with unstructured data completes the picture for the users.

Now let's see why the Enterprise Search in its nascent stage

• Market still has to wake-up to its potential, and Vendors have yet not offered effective, widely acceptable and convenient offerings.
• The enterprise search and Business Intelligence players are different specializations, and they still have to actively engage with each other.
• Frantic consolidation phase, leading to an eye off the innovation, with more focus on integration of the acquisitions.

The barriers to growth of enterprise search:

The enterprise search is more of a business and modeling challenge than a technology challenge. For an enterprise search to be effective, we need to have the following enablers, which will take time to achieve value add.

• A robust and well-defined meta-data model, and the user search needs to be guided by the meta-data model. Just imagine, if there are 100 different reports having hundred different terms of revenue. A user will see "Net Revenue" figures as being different across multiple reports, because inherent definition of "Net Revenue" may be different in these reports. Building a robust meta-data repository could be a big issue, as most of the organization struggle with that.
• Guided discovery of search: The Business Intelligence search will become effective, if there is a guided search mechanism in the search capabilities. This will mean that a person when he gets 100 reports as the search results, he should be able to provide intelligence clues to home onto the right report which will provide the data. This is different from adding some additional keywords in your search bar.
• Published reports will most-probably need to have "search tags" to ensure a faster and effective search. This will need technical changes in the products and also a more informed super-users.

Lets explore different ways including of using SOA to address the above barriers in my next post.

In the last post I had given a primer on service oriented data warehousing. I would like to dwelve deeper into some of the topics that SOA and DW holds in parlance.

One of the most important topics is “DATA”, where DW community had learned valuable lessons over years of experience in data warehouse implementations. The same is case with SOA, where pioneers of SOA have learned valuable lessons on data quality, data integration, and governance.

Any SOA initiatives will fail:

• When they sit on a platform of poor quality data,
• Attempt to share data that is not integrated across the enterprise,
• Implemented with insufficient thought given to security, compliance, and change management.

It strikes to me about “abstract services” in the data warehouse world, which are commonly recognized, simply stated, and independent of specific data sources, business processes, and BI deployments. I guess we can do that, let’s see how this works.

Consider the relationship between the dimension manager and the fact provider.

A dimension manager is the centralized resource for defining and publishing a conformed dimension to the rest of the enterprise. A master data management (MDM) resource is an ideal candidate for dimension manager. Enterprises where MDM is not instititionalized, the data warehouse team is a kind of “downstream MDM” function that gathers in-compatible descriptions of an entity such as customer and publishes the cleaned, conformed, as well as de-duplicated dimension to the rest of the data warehouse community.

The subscribers to this dimension are almost always owners of fact tables who want to attach this high quality conformed dimension to their fact tables so that BI tools around the enterprise can perform drill across reports on the conformed contents of the dimension.

There are two basic services that every dimension manager publisher needs to provide to the fact table subscribers.

Fetch means that fact table provider pulls information from the
dimension manager.

Alert means that the dimension manager pushes
information to the fact providers.

Let me elaborate;

• Fetch specific dimension member (we assume in this and the following steps that thedimension record has a surrogate primary key, a dimension version number, and that the information transmitted is consistent with the security and privacy privileges of the requester).
• Fetch all dimension members.
• Fetch dimension members changed since specific date-time with designated SCD Types 1, 2, and 3.
• Fetch natural key to surrogate key correspondence table unique to fact table provider.
• Alert providers of new dimension release. (A major dimension release requires providers to update their dimensions because Type 1 or Type 3 changes have been made to selected attributes).
• Alert providers to late arriving dimension members. (Requires fact table provider tooverwrite selected foreign keys in fact tables).
  
  These services are generic to all integrated data warehouses.
  
  In a dimensionally modeled data warehouse, we can describe the administrative processing steps with great specificity, without regard to the underlying subject matter of the fact or dimension tables.

That is why, in SOA parlance, dimensional modeling provide,s well-defined reference architecture on which to base these services. These services seem pretty defensible as meeting SOA design requirements.

SODW Success is Enterprise Success

Mantra for Successful SODW

1. A focus on the nouns and the verbs of your organization to provide guidance in decision making when contemplating new projects. Does the proposed project fit within and/or adhere to our organizational definitions and requirements of nouns (data) and verbs (function)? This new system must integrate with the rest of the enterprise.

2. Data governance programs and policies to define clear ownership, purpose, and use of data in the organization.

3. Function governance programs and policies to define clear ownership, purpose and use of shared functions and services in the organization. As data management professionals, we all know the importance of data governance, so we are all prepared for making sure our functional architecture has the same discipline and rigor.

4. A Master Data Management (MDM) strategy that supports the integration of and use of the shared data in the organization. MDM, in my mind, is one of the key components to any effective service oriented architecture (SOA), and often gets overlooked. You cannot effectively share services if you do not effectively share the data that is the backbone of your organization.

5. Data integration and data access services are the core services that allow us to integrate data into our MDM solution and warehouse, as well as provide common access to that data. This is where we make data integration lighter and provide the ability to access analytic data more dynamically with other, non-analytic processing. This is also where we can integrate the rest of our organization’s data, the semi and un-structured data.

6. Data quality and standardization services to make sure that the shared data being integrated into the MDM solutions are all applying the same rules. In fact, it would be best to have these services pushed as close as possible to where the data enters the organization, whether it is typed in on a web page or comes in as an XML transaction from a 3rd party. Many data warehouse solutions do the data quality and standardization; but if I can leverage these as services and put them at the point of entry then we will have better data across the organization, making integration easier and more cost effective.

7. A technology platform that facilitates the deployment and orchestration of data services as part of every day business processes. Something like “closing the loop” in data warehousing. This is where it becomes possible to leverage data and functionality that was exclusive to the data warehouse and business intelligence applications and make them part and parcel of everyday business processing.

To be successful in deploying a SODW, your organization must realize the increased interdependency with the rest of the applications in your organization, which is a good thing. Previously we moaned about the hassles with poor data governance upstream from the data warehouse. Well, now we also have to worry about having a consistent, interoperable function architecture as well. Again, in my mind this is a good thing. It is an opportunity, initially borne out of the need to integrate data, and now function, that will push the data warehouse to the center of the organization.

To conclude the data warehouse is no longer a proxy for the enterprise data architecture, but a driver for better overall enterprise architecture which is sharing, reusing, standardizing, increasing efficiency… Sounds an awful lot like saving money to me.

Service Oreinted Data Warehousing - SODW

When I sat down to write this blog on service oriented data warehousing (SODW), I had to reflect on the work and struggles I have had in data warehousing and data integration, the people I had learned from, and some of the failures trying to push these concepts before they were technically feasible. From those reflections I came to realize that

1) a data warehouse is a proxy for good enterprise data architecture, and

2) the process for which these things are being built makes them very difficult to change as the needs of the organization changes. Not to mention that most organizations have separate groups that do operational integration and data warehouse integration (ETL).

Data warehouses, to some extent, have become part of the problem they intended to correct: another system that demands resources for its care and feeding, yet doesn’t really integrate well with the other systems.

How can we build solutions that allow us to integrate data for both operational and analytic purposes?

If we can integrate data more easily, would that not facilitate operational analytics and active data warehousing?

How do we integrate and tap into all that semi-structured and unstructured data?

To satisfy these questions, to really close the loop and make the warehouse part of the organizations daily activity, is to rethink how we integrate data and expose that data to the organization.

I don’t want to state the obvious here but many data warehousing and business intelligence projects are not as successful as they could be because they lack a sufficient enterprise foundation. There are many things that must be in place such as data that is standardized and of a high quality, consistent definitions of information and function, and ownership of data and function.There is an important point I need to make here: the need for a solid foundation doesn’t change just because we are throwing around the “service oriented” tag. Let’s not confuse the need for flexibility and agility with quick and dirty. SODW still requires good enterprise data architecture practices that are aligned with how the data is used. Perhaps even more discipline and rigor are required than we have seen in the past. In fact, a successful SODW program will help drive improvement by focusing on doing things well once and reusing them, instead of doing the same things many times quickly. In the long run, reusability is more cost effective and more agile.

So what do we need to for successful SODW? Really, we need the same things across the enterprise to make things truly interoperable and reusable. These things are not only keys for successful SODW but, in my opinion, successful enterprise applications. I will explain them in my next post