A simple answer is – data that is big. Obviously the reader, viewer or the listener, if they are patient enough would pause to ask – “what is big?” In most cases they would have flipped the page or clicked the back button or changed the channel.
This is an ever-changing phenomenon. Depending on whom or when you pose the question to, you would end up with a different answer. Big data can be big for different people. In data processing a limitation can be reached in many ways. One can reach a storage limitation and find that a new technology is required to efficiently store data. Assuming that enough storage space is available one may perceive a limitation is ability to process the data. Some yet may experience a limitation in arriving at a meaningful result after processing the data. There a few who are able to store, process, arrive at a meaningful results but are unable to present the results in a form that can be easily understood, digested and disseminated.
Data hence can be big because of three broad reasons.
- The data is inherently large – this would be the case where we are having say the data of all the visible universe in the sky
- The data is simple in the atomic sense but are very large when seen together – a mobile call characterized by a source and a destination number is simple but all the calls in day made by all the people in the world is very large.
- The rate at which data is generated is so high that it is impossible to make meaning at the same rate as it is generated – say the data generated by the Large Hadron Collider at CERN – so huge that it would take relatively long time to make any meaning of the same. The challenge here is to store the generated data and later condition them for any sort of analysis.
In large corporate environment there is an awakening of sorts that is leading to the belief that if such large data is available then there must be some hidden truth inside it and it is there for anyone willing to make the effort to extract the same. I am reminded of the secret ingredient in the movie Kung Fu Panda. How Tai Lung was bent on attaining the secret ingredient for himself!
Does Big data hold such a promise? Technologists can only offer better tools and methods to manipulate, massage, re-condition the data and make it available for the business. Will the business recognize the pot of gold when offered on a plate with watercress around it? (Sorry Pelham, I could not resist that!). Business people on the other hand are hampered by not knowing the capability of the current technology and its capability. Someone has to put to them a question that may go something on the following lines – “If we can extract the customer buying patterns and match it with the weekly football games then we believe we can plot the beer consumption against the clubs that are playing every weekend. Will that help your stocking policy in the beer section?” Then a business person may perk up and say “That would be great! Then on days when ManU is playing we may push 20% more to the areas of the opposition because we believe that they would be drowning their sorrows in our beer”.
A bit of levity there but the reader gets my drift.
The next big question is then, what are the tools and skills sets one would require to be that someone?