Data is the New Oil

The phrase “Data is the new oil” has been the recent topic of discussion amongst the business executives. Data is the genesis for insights but the question is how much value do we really extract from it? Extracting value from data has indeed created profitable avenues, just as it has from oil. The oil metaphor for data is interesting and needs to be taken a step further. When put through a data-refinery, especially in real-time or near real-time, it provides invaluable Operational Intelligence (OI).

OI answers the question, “What’s happening within my business and IT operations now?” OI platforms do this by accumulating mountains of data, refining it, and overlaying it with machine learning and analytical models to extract insights in the form of live dashboards, reports and real-time alerts (see Table 1: Data Types and their outcomes). This output is invaluable for executives of large and medium size enterprises tasked with dynamic, real-time decision making. For them, real-time updates and continuous insights have become imperative. This data can be used in scenarios for answering questions such as “Should I procure additional servers now or delay the purchase to next quarter?” or “Am I about to miss my SLA for the next 200 customers? What action should I take to mitigate the fallout?”

Data Types and their Outcome

Data Input Types 

Offline or slow moving data: Daily, weekly and monthly data from business systems (ERP, CRM,SCM, HRM) and IT inventory. (Asset, Capacity, external data).

Near real-time data: Hourly data from servers, storage, router, networks, websites, portals,device telemetry.

Real-time data: Continuous data stream from machines, routers, transactional systems, events, etc.

Data Output

Various input data are correlated together to give OI in the form of reports, dashboards, live monitoring, alerts and actions. It includes:

• System failure detection
• Auto-ticketing and actions
• Security threats detection
•  System availability
• System performance
•  System scale-out/in
• Capacity forecasting
•  Consumption and chargeback report
•  Trend analysis and forecast
•  SLA adherence

Enterprises need all three data types – historic offline or slow moving data, near real-time data and real-time data – to arrive at sophisticated cross-relations resulting in pro-active diagnostics, event predictions, and recommendations that enable quick and immediate decision-making. The most significant driver for OI adoption is its ability to deal with a dynamic world moving at Google Speed. Decisions can’t wait for weeks now; we have minutes or often seconds to take them. For instance, in an e-commerce website, a delay in page loading time by a few seconds results in customer attrition. If there was to be a real-time event correlation (page load times and the number of abandoned carts), it would help resolve the problem in real-time before significant revenue loss.

The Data “Refinery”

Managing and transforming data into OI is a key remit of CIOs in medium and large enterprises. Their goal is to keep an (digital) eye on business and IT activities so that aberrations, inefficiencies and threats can be dynamically identified along with prescriptive solutions before they impact operations. Data, and how it is captured, stored and correlated, becomes critical to achieve this goal.

Sources of data - the new oil wells:

Any business or IT operation has several sources of data (Table 1) and the challenge therefore is to rapidly transform the input from multiple and varied sources into desired business and IT operational insights.

Moving the data through the “refinery”

The “refinery” must ingest and move all types of data to appropriate containers, stores and models. A variety of tools for data collection (e.g. Scoop, Flume) que the data for persistent storage leveraging real-time data streaming technologies and object/file stores. Once the data is inside the `refinery’, data processing, search and analytics models and tools are applied (e.g. real-time processing, machine learning for predictive modelling, batch processing). The ‘refinery’, in essence, transforms the data.

The finished data “refinery” products

The “refinery” is the insights engine that uses data to extract value and spew out intelligence, action and visualization in formats that operations can intuitively use for accurate decision-making. Operational Intelligence can take the form of authentication and authorization, predictive analytics that preclude threats and system failures, auto-ticketing, availability and capacity forecasting, chargebacks to LoBs and SLA adherence.

The efficiency and accuracy of the digital insights engine depends on how the data “pipes” are laid within the “refinery” and how its components are integrated and configured. This determines the manner in which data is ingested, orchestrated and crunched. The architecture and components within the insights engine determines the types of data an organization can leverage. Ideally, this should range from offline to near real-time to real-time and must include structured, semi-structured and unstructured.

Conclusion

OiI platforms’ abilities to operate in scalable environment, to quickly re-configure itself using accelerators and best practices, to lay the data “pipes” and analytical models in new combinations and structures help enterprises manage their data across its lifecycle and extract from it the fuel that they require to sustain their competitive edge in an ever changing landscape.

Balwant Singh

Balwant Singh is Practice Head, Data Platform Engineering and Senior Member of DMTS (Distinguished Member of Technical Staff) Group at Wipro. Balwant is a technologist with passion and drive for emerging technology business with more than 17.5 years of experience in Software Products and Internet Scale Solution engineering of highly available, fast, reliable and scalable systems using open source and disruptive technologies such as Cloud, Big Data/Hadoop, HPC and Cluster Computing & Storage Systems.

Balwant holds a Bachelor of Technology Degree in Electronics & Communication from the University of Pondicherry, India.