Achieving success in Pervasive BI

Pervasive business intelligence (BI) is the latest trend in enterprise data warehousing (EDW) and BI around the world.  In simple terms, pervasive BI delivers data warehouse insights to everyone in the enterprise, not just the privileged back-office knowledge workers.

Its goal is to make thousands of daily tactical decisions better by using facts instead of experience and gut feel alone.  Imagine where every employee making 40-50 small decisions a day.  Also imagine adding pervasive BI so that 20 of those decisions are supported by facts instead of guesswork. Generally, the line-of-business managers understand pervasive BI instantly and wonder why they haven't had it all along. 

Unfortunately, computer technologies were limited, forming a barrier between the front line user and the EDW data until around 2001.  It existed because front line users need "fresh data" in addition to the historical information normally found in the EDW.  While historical data is loaded into the EDW nightly or weekly, front line employees need today's data as well. 

For example, a call center representative (CCR) needs to know a lot about the consumer calling in: residence, prior calls, profitability score, and the next best offer to suggest.  All this can be calculated nightly, ready to go each morning.  But the CCR also needs to know what calls the consumer made this morning and what the consumer did on the company website today.  Fresh and historical data need to be displayed while the consumer is still on the phone.  Getting that data from the production system, into the data warehouse, and back to the front lines is sometimes called "real time BI".  Until 2001-2002, the software and computers could not do this.

As often occurs, some enterprises didn't care about technology.  They wanted the benefits of pervasive BI now.  So in 2001-2002, numerous enterprises found ways to speed up getting data into and out of the EDW.  These visionaries were able to connect their point of sale machines, airline gate agents, web site, call center, and dozens of other front line users straight into the EDW.   As IT organisations charged ahead, the software vendors saw opportunities to improve their products in the same direction.  In a symbiotic relationship, IT experts and vendors collaborated to make the technologies work in real time.

What were the barriers that were overcome?  First, three technology subsystems had to evolve.  Second, IT organizations needed new designs and processes. 

The three technology subsystems - Data Integration Services, Decision Repositories, and Decision Services - are part of an information supply chain.  Data starts in raw form, gets transformed and cleansed, then it's repackaged and stored and repackaged, and finally analyzed and distributed.    

Data Integration Services is a hub, taking all kinds of data from many production applications, and preparing it for the EDW.  To meet the service level goal of "under one hour" or "under five minutes," delivery of the raw data needs to be "guaranteed delivery" to the data integration hub so nothing is lost and everything is on time. 

Data integration services also must clean up the data quickly or inaccurate data will lead to bad decisions.  It must remove duplicates, parse name and address, and swap out "codes" for more understandable values.  All this must occur as multiple streams of data flow into the data integration server continuously or in spurts. 

The decision repositories (data warehouses and marts) need to load the data and serve it up as fast as possible.  The task is difficult because data loading consumes the entire EDW server, leaving nothing for the front-line and back office employees. Performance for the front line and back office users is terrible during loading. 

That's when the concept of real time data loading utility and mixed workload management tools began making an appearance.   Mixed workload management means prioritising work inside the data warehouse according to business rules.  This enables an enterprise to give the call center workers top priority, reports medium priority, and data loading low priority.  The result is data loading tasks perform well while reports run fast, and the call center runs fastest.  

Decision services grabs the EDW data, reformats it, analyses it, and delivers it to the front line user.  Since the front line user doesn't have time to read reports, the requested facts have to be short and specific to the business task.  This means delivering analytic information to BI dashboards, mobile devices, employee portals, and modern "composite applications".  Since nearly 50 per cent of all IT development projects are now composite applications, the EDW and decision services must fit into those developer tools and applications.  The result is the ability to deliver analytic insights in any business process.  Pervasive BI.

The IT organisation also has some critical success factors.  First, there must be a service level agreement between the IT Division and the front line users for performance and availability.  Negotiating the agreement wrings out all the cost issues, true business needs, and project objectives.  The second success factor is mission critical availability. 

The entire information supply chain must be "always on, always available."   Since many business processes run any time of the day, the pervasive BI subsystems must as well.  But most companies have not made the end-to-end information supply chain mission critical.  This is the area of the biggest risk.  The good news is most IT divisions know how to do it with their mainframes and UNIX servers.  They simply need to apply those principles to the pervasive BI infrastructure. 

Pervasive BI is trending upwards, having moved from the early adopters into the mainstream.  The visionaries have blazed a path but there is still time to be a leader in your market using pervasive BI.  Just don't wait until this trend is, well, pervasive.