In recent months, we’ve been hearing about the other 99%, but earlier this year Forrester began discussing the other 95%. That’s the massive “ever larger mountains of data produced by our cellphones, computers, digital cameras, RFID readers, smart meters and GPS devices” (via NYT). Big data represents that massive data set companies fail to effectively sort and utilize. Business Intelligence (BI) mines about 5% of the available data for businesses. To apply the BI model of gathering and analysis to the other 95% of big data is cost prohibitive. The amount of data is simply overwhelming.
If a company can effectively utilize just some of that untapped 95% over and above their competitors, they have the potential to optimize certain aspects of their business and increase the likelihood of capturing their market. Forrester has suggested that 2012 will be a time for exploring the possibilities of tapping the other 95%. Recently, the New York Times explored this burgeoning field of big data, and has suggested that it may open the door for new players in the data analysis field.
While companies like Oracle and I.B.M. have dominated the data analysis field, new players like Splunk are emerging who offer promises of affordable solutions for beginning to tap the mountain of big data. Servicing explosive companies like Salesforce.com and LinkedIn, Splunk offers a new approach to thinking about machine data and operational intelligence.
Forrester suggests three trends toward exploring the other 95% in the coming year:
1. The Growth of Hadoop
Originally developed by Google as a means of indexing the varied types of information they collected to help them offer meaningful and useful results to users. Hadoop spun off as an open source project for yielding meaningful data from large clusters of complex and structured information. Forrester suggests that many companies are ready to move forward with enterprise-level Hadoop initiatives in the coming year. So expect the number of Hadoop deployments to rise.
2. Dramatic Growth of Analytics
Expect a wide range of vendors to emerge with a primary focus on intelligence and analysis centered on large data sets. Due to the nature of big data, this will require new ways of thinking about how analysis and intelligence works. Steve Lohr at NYT has suggested that the real challenge is not in software but in people. This means the rise of data scientists who can look for patterns in the “noise of vast unstructured data sets.”
3. Graphing Databases
Big data reveals the need to visualize relationships, influence and behavioral propensities. “Social graph analysis” will become a prestigious and highly skill set that addresses the need for mining the rich textural data available.
This emerging field of big data sorting and analysis will impact how IT dollars are spent as companies continue to look for ways of moving IT from cost center to profit center. As this field emerges, Integracon will continue to discuss and work with our clients who are seeking to learn more about how to adapt the rich possibilities of this emerging science in their business.
[...] in the week I wrote about the great potential for leveraging the untapped 95% of information. Forrester calls this “big data” and suggests that effectively utilizing the vast [...]
Pingback by Exploiting Technology For Competitive Advantage « Integracon Technologies Blog – Computer Support in Knoxville, TN — January 1, 2012 @ 9:51 pm