This is the year big data becomes an integrated part of an agile business strategy. That’s not just a function of technology, but of culture and process. It all begins with the realization that extracting value from big data depends on adapting to a constantly changing environment around us.
Big Data Doesn’t Stand on its Own
Oliver Ratzesberger, Senior Vice President of Software at TeradataTDC +0.00%, observed, “we are maturing in big data,” as companies come to realize that just putting up big data doesn’t cut it. “Without business results, big data is just a science project,” Ratzesberger said. That’s the mistake many businesses made when they jumped on the big data bandwagon without a plan for its use.
Doing big data wrong
For most businesses using big data meant setting up monolithic databases that required lead times of 18 months, a time frame that hurts agility. To get around that, users would set up individual “data marts” that fragment and duplicate information to the point where the whole organization can lurch into a wasteful “data anarchy,” what Ratzesberger calls the “wild, wild west,” beset by inconsistent information that doesn’t give the answers businesses need.
The way forward
As a Gartner press release states, “the value is in the answers, not the data.” That’s why this year, according to vice president and Gartner Fellow David Cearley, the spotlight is on the efficient retrieval, analysis and delivery of the data-driven answers. “Organizations need to manage how best to filter the huge amounts of data coming from the Internet of things, social media and wearable devices, and then deliver exactly the right information to the right person, at the right time.”
The Culture of Agility
Startups often have an advantage in achieving agility because they don’t have to work around bureaucracy and inflexible systems. But why should large organizations be stuck with ‘slowness?’ In the big data era, even the largest, global companies that do have process restrictions can find pockets of agility. Small steps can lead to the agile mindset spreading like a virus throughout a large organization.
Ratzesberger said that even banks, which are subject to the most regulated environment because they have to provide audit trails, can pull it off. He offered the example of Wells Fargo, which has successfully integrated agile big data around marketing and direct contact with the consumer by shifting from transactional thinking to behavioral analytics, combining Enterprise Data Warehouses and Hadoop clusters to build a unified analytics platform in the process. Organizations that want to make the most of big data need to embrace this startup mindset of adaptation.
The Challenge of Repeatable Results
The real trick is not just getting answers that work in the short-term, but in achieving repeatable results. Even the most advanced big data companies like Amazon, Google and eBay have difficulty with repeatability of big data results due to changes in key variables.
Ever get a promotional email or ad that has no relevance to you? We all have, and it’s usually due to the marketing algorithms used to analyze big data inputs responding incorrectly to the wrong signal. For example, eBay started applying algorithms to the tags used to track customers in 2007 to measure the relevance of search results on its site. After a couple of years of success, the results became less accurate and seemed more random and arbitrary. The algorithms no longer worked because one of the tags had shifted. Events like that one resulted in customers seeing search results or receiving marketing emails that made no sense to them.
“The algorithm is not a human brain and doesn’t realize that the parameters have changed when tags change,” Ratzesberger observed. If a change is made to a variable, everything “downstream” from that variable must change, too, or the complex results can backfire.
The Sentient Solution
The solution to this entire problem of achieving agility at scale is the Sentient Enterprise, a concept that Ratzesberger developed with Dr. Mohan Sawhney, a professor at Kellogg School of Management at Northwestern University. The Sentient Enterpise is capable of self-awareness and real-time responses to changes in its environment. It combines technology, data and automated decisions to eliminate the obstacles to agility posed by data silos, data drift and the delays that prevent real-time responsiveness. The Sentient Enterprise system picks up on changes that require redirection for algorithms to deliver more accurate results.
As businesses understand that a static system doesn’t work for agility or long- term results, they are adopting the culture and processes needed to make 2015 the year of sustainable and agile big data.
Ariella Brown is a freelance writer who specializes in writing about technology, including big data, analytics, social media, and their application to education, health, business, marketing, science, and society.