Big Data The Next Big Thing

Rohan Murdeshwar highlights the crucial role that ‘Big Data’ plays in our everyday lives

The French city of Bordeaux is often associated with expansive verdant vineyards, beautiful churches and some of the finest art galleries. Introduced by the Romans in the 1st century, Bordeaux’s wine has earned it a famed reputation across the world – a reputation that remains untarnished twelve hundred years later. However, in the late 1980s, wine connoisseurs sensed that a rather unconventional force was threatening their informational monopoly on wine quality. Orley Ashenfelter, a Princeton economist, was using statistics and data analysis to predict the price and quality of fine Bordeaux wines, a task until then only bestowed upon the refined palette of the world’s wine experts. The fact that the weather was a strong influence on wine quality was not unknown, however, Ashenfelter did what any number cruncher would. He amassed gigabytes of weather data from the Bordeaux region and discovered that the finest wines were harvested during periods of low rainfall and high summer temperatures. The outcome of his analysis was a statistical equation, which on inputting weather data of any year would tell you the quality and price of that year’s wine. His two-line equation turned out to be a far more accurate forecaster of wine prices than the taste buds of seasoned wine connoisseurs.

From Wine to Web Browsers
Ashenfelter’s experiment is just one of the myriad examples where large datasets and mathematical models are being used to offer better predictions and improve decisionmaking. Walmart, the world’s largest retailer, uses data analytics to determine on which shelf its products should be placed so as to maximise revenue. Amazon, the world’s largest e-commerce firm, utilises its customers’ online browsing habits to develop algorithms that offer tailormade product recommendations on its website. Google uses its treasure trove of accumulated data to develop models that learn from users and offer more relevant search results. Big Data is used in times of human crisis as well. Epidemiologists use call-data records (CDRs) generated by mobilephone calls to contain the spread of diseases such as the recent Ebola outbreak in West Africa and Mexico’s swine-flu epidemic in 2009. Sports teams use data analytics to develop team strategies and improve player performance, as portrayed in the film Moneyball. On analysing data from social networks, credit card companies found a high correlation between credit card defaults and customers who typed their Facebook names in uppercase letters. From governments to airlines to rental car companies, insurance firms and banks, big data is transforming how we understand the world, do business and implement public policy.

What is big data?
It’s a term used to describe the exponential growth and prevalence of structured and unstructured data from which one can glean information. Big data and data analytics are often used interchangeably however, there are differences. In 2001 Doug Laney, now an analyst at advisory firm Gartner, defined big data as “high-volume, -velocity and -variety information assets that demand cost-effective, innovative forms of information processing for enhanced insight and decision-making” now commonly known as the ‘three Vs of big data’. Walmart collects approximately 3.5 petabytes (that’s 3,500 1 terabyte (TB) hard disks) of data every hour from its customers’ transactions. Globally, 2.3 zettabytes (equivalent to 2.3 billion 1 TB hard disks) of data are created every day! The velocity of data generated today enables Google to study real-time data and inform its users of traffic congestion and blocked routes. Big data accumulated from a variety of sources such as social networks, mobile phones and sensors helps consumer product companies get an unprecedented view into its customers behaviour and purchasing preferences. Additionally, rapidly declining costs of computation – processing power, bandwidth and storage among others – has made it far more economical to analyse large datasets.

Adapt or perish?
The disruptive power of the Internet, technology and big data is causing an increasingly recurring trend. Organisations from the pre-internet period are being rendered obsolete by those that are born in the digital age. Netflix, the US based online movie-streaming provider, caused Blockbuster, a home movie rental service, to file for bankruptcy. Amazon put Borders, an international book and music retailer out of business. However, all traditional brick-and-mortar establishments are not destined for the grave. The ones who adapt to the digital age and incorporate data analysis in their business strategy can gain competitive advantages over their offline business competitors. Indeed the most vital aspect of big data is its “enabler as an enhanced decision-making tool, offering a far superior alternative to ‘intuition’ in corporate boardrooms”, say Rajiv Sharma, the VP at an analytics consultancy based in Gurgaon. This does not mean that human knowledge and industry experience have become irrelevant. On the contrary, determining which qustions need to be addressed and what problem need to be tackled using data, requires critical human insight.

Pebbles Build Mountains
One should also be conscious of the fact that big data is not a panacea for all the world’s problems, nor does it result in massive instantaneous gains. Tiny incremental tweaks spread across a large customer base and over a long period of time aggregate to produce cumulative effects that are immense. Take for instance UPS, the world’s largest shipment and logistics company, whose engineers found that left-hand turns against traffic wasted precious time and expensive fuel. By adjusting the GPS trackers in its vans to make only right turns it saved approximately 10 million gallons of fuel and significantly reduced emissions. Big Data Wins Elections Nate Silver, a statistician, former poker player and writer for the New York Times and current editor-in-chief of ESPN’s FiveThirtyEight blog gained instant stardom when he accurately predicted the outcome of America’s 2012 presidential election. By building sophisticated statistical models, he correctly forecasted the way each of America’s 50 states voted proving yet again that number crunching can trump political pundits’ ‘gut feelings’. Closer home, Today’s Chanakya, one of India’s leading public opinion polling companies, was the only polling agency to get close to the National Democratic precious time and expensive fuel. By adjusting the GPS trackers in its vans to make only right turns it saved approximately 10 million gallons of fuel and significantly reduced emissions.

Big Data Wins Elections
Nate Silver, a statistician, former poker player and writer for the New York Times and current editor-in-chief of ESPN’s FiveThirtyEight blog gained instant stardom when he accurately predicted the outcome of America’s 2012 presidential election. By building sophisticated statistical models, he correctly forecasted the way each of America’s 50 states voted proving yet again that number crunching can trump political pundits’ ‘gut feelings’. Closer home, Today’s Chanakya, one of India’s leading public opinion polling companies, was the only polling agency to get close to the National Democratic Alliance’s (NDA) final tally of 336 seats (it predicted 340 seats for the coalition). It was the only agency to use a Bayesian forecasting model, one that uses prior knowledge of a population and then updates it with interviews from a random sample of voters to arrive at a final opinion. The Bhartiya Janata Party’s (BJP) resounding success in the 2014 elections can, at least in part, be attributed to its adoption of big data analytics. Narendra Modi’s pervasive social media presence laid the platform for the BJP’s data-centric strategy. In a country with over 900 million mobile phones, it used phone data to create algorithms that helped it determine voter demographics such as age, location and income levels. This insight was the sharp end of the stick that it used to carefully target voters and build a successful campaign.

Big Data = Big Brother?
With Big Data becoming ubiquitous and technology enabling corporations to continuously gather more data from users, today’s information landscape is beginning to wear a rather Orwellian look. The battle for data privacy is on going with every smartphone user being a soldier on the front lines. Companies use technologies such as cookies and e-tags to identify users as they browse the web collecting information such as what users shopped for and where they live. From this, they can infer other personal details such as age, gender, and annual income and often users are unaware that they are being tracked. Companies insist that they don’t have information of user names but possess only numbers in a database. Analysis methods such as ‘homomorphic encryption’ enable companies to process data while it is in an encrypted state thus protecting user privacy. However, performing computations on encrypted data takes about ten thousand times longer than open data and is considerably more expensive. Moreover, industry experts believe that no data are perfectly encrypted and linkages to the end user can be easily ascertained. Some believe that more government intervention is required in order to draft regulation that forces firms to comply with data privacy laws. However, Edward Snowden’s revelations that America’s National Security Agency (NSA) was and is running a surveillance programme called PRISM to collect information about its citizens from big technology firms, proves that governments themselves are very much in the data-gathering business. On the other side of the debate are those who believe that data privacy concerns are overstated and having their personal details collected is just a small price that users pay for more relevant and customised information.

‘ The Sexiest Job of the Century’
The Harvard Business Review touted the job of a data scientist as the ‘sexiest job of the 21st century’ with some comparing them to Wall Street’s quants of the 1990s. “Many industrial sectors are seeing a severe shortage of people trained in mathematics and statistics”, says a professor at an Ivy League university in America. Top educational institutions, both in India and abroad, have launched data analytics programmes to cater to a rising demand for the skill set that firms are specifically looking for. With demand far exceeding supply, Indian technology firms such as Snapdeal, Flipkart and Amazon are continuously on the lookout for data scientists luring them with lucrative job offers. Smit Mehta, a Trainee Decision Scientist at India’s largest analytics provider in Bangalore says that a job in data science is ‘pretty awesome’ and one that ‘lets you interact with decision makers on a regular basis’ where ‘analytical insights are actually implemented by the company’.
Big Data is still in its infancy in India and even around the world. However, going by its importance and usefulness, organisations and people acorss the globe are only now realising its immense power. From something as innocuous as wine tasting to more pressing matters such as the Ebola crisis and elections in different parts of the world, Big Data has the potential to drive better decisions, help strengthen governments and change the world along the way.

As interest in Big Data is increasing exponentially, the number of job opportunities are increasing as well. For example, McKinsey has calculated that in North America alone, companies will need around 1.5 million managers and analysts with the know-how to use the analysis of Big Data to make effective decisions in the next five years. While Gartner has predicted that more than four million jobs will be created by 2015. Students graduating from a Master in Business Analytics and Big Data will have many different opportunities as any enterprise that needs to optimise its decision making process and improve its performance to be competitive will need this kind of professionals. Some of the exciting professional activities that graduates from this field can do are smart cities, management consulting, marketing intelligence, energy analytics, logistics and supply chain optimisation and financial analytics just to name a few.
– Marcos Ramírez Amoretti Associate Director, Business Analytics & Big Data, IE, Madrid 

UNIVERSITY COURSES IN BIG DATA/ANALYTICS
▶Certificate Programme in Business Analytics, Indian School of Business (ISB), Hyderabad
▶Certificate Programme in Business Analytics, IIM Lucknow
▶Executive Programme in Business Analytics, IIM Calcutta
▶Advanced Analytics for Management, IIM Ahmedabad
▶Master of Science in Business Analytics, National University of Singapore (NUS), Singapore
▶Master of Information Systems Management, Business Intelligence and Data Analytics (MISM-BIDA), Carnegie Mellon University, USA
▶Master of Science in Analytics, Northwestern University, USA
▶Master of Science in Analytics, North Carolina State University, USA
▶Master of Science in Business Analytics, New York University Stern School of Business, USA
▶Master of Science in Analytics, University of Chicago, USA

 

Volume 4 Issue 7