by Daniel Graham, Teradata
It’s one thing for a forward-thinking CEO to want his organization to make more extensive use of analytics. It’s quite another to overcome the challenge of getting more analytics out of the existing pool of resources. Programmers can only produce so much code. There are only so many professional services that can reasonably be bought to tackle big analytics projects.
There is in fact a tremendous, untapped solution to those limitations and it lies not with hiring more programmers and professional services, but instead in crowdsourcing. It turns out that there are many people developing apps and processes outside IT. And most of those apps get lost or permanently hidden on one laptop. That’s intellectual capital down the drain.
Empowering business people requires giving them a different breed of tool. They need a place that can provide them with quick access to a toolbox of pre-built algorithms and functions, a portal of sorts for micro-services. Like an app store for analytics, they can search through offerings and put together a kit of connectable analytic “apps” to create a workflow that will let them extract the data they need from a file system, pass that data through some data prep tools, send the resulting data through analytics, and then observe the data through visualization tools. They are in effect building analytics programs out of these micro services, without needing programming skills or even the assistance of a programmer.
Achieving this will require building up a repository of analytics components. Right now, when professional services are hired, it’s like bringing in master craftsmen to create a bespoke item, which is awesome for its intended purpose…and that’s it. It’s used in a particular way and for nothing else. And if its utility should run out, there’s not much value left in it.
It would be more effective if every time significant work is done on analytics, not only is the immediate issue solved, but something is added to the institutional storehouse of techniques and tools that can be applied to future problems. If instead those professional services had been applied to creating a palette of micro-services, users could reconfigure them in endless and even unpredictable ways.
This is crowdsourcing analytics: from the business users themselves (they write scripts and adjust everything IT gives them), from vendor app stores (data prep, attribution, scoring), grad students (experimental math), and contractors. And it all needs to fit into a portal-like framework for easy access, as easy as getting apps for your smartphone.
Steps toward crowdsourcing analytics can already be seen. Several products are already on the path to an iTunes-like approach to self-service analytics for business.
The benefits to this approach are two-fold. The first is in the economics of reuse, putting programming assets to constant repurposing in new and beneficial ways. And the hidden benefit is the means to access untapped creative potential that currently lies dormant in nontechnical people who nonetheless may have very keen insights into analytics that impact the business.
There are legitimate obstacles to a proper marketplace, including concerns about standardization of APIs and even the data itself. Until now, all the players have been off doing their own thing. But as algorithms become readily available commodities and more companies create marketplaces to enable micro services to accelerate exploration, there is increased pressure to be sure that data and these apps will play nicely together.
More interesting are the potential benefits for micro-services presented in an easy-to-use marketplace. Organizations will become more nimble. When a manager wants a recommendation engine to drive sales, it won’t all need to be built from scratch after weeks or months of development. We can no longer afford to wait days or weeks for IT developers to deliver every imaginable business need. We must tap into crowdsourcing to create an open marketplace where analytic micro-services can be shared in much the same way that we get apps for our smart phones from the App Store or Google Play.