Ting Hsuan Lin | September 27, 2016
The two biggest trends in the data center today are cloud computing and big data. This writing examines the intersection of these. For consumers, big data is about using large datasets from new or diverse sources to provide meaningful and actionable information about how everything in the world works. For example, Netflix can use customer data to produce that shows tailored to their audiences. For producers, however, big data is about the technology necessary to handle these large, diverse datasets. Producers characterize big data in terms of volume, variety, and velocity. How much data is there, of what types, and how quickly can you derive value from it?
The macro trends behind big data
Just as adopting a service-oriented approach is the macro trend behind the cloud, there are several macro trends behind big data.
The relationship between Big data and Cloud
So what is the relationship between big data and the cloud? Big data has its origins in the cloud. Apache Hadoop, one of the most widely used big data technologies today, was built on research from Google and initially deployed at Yahoo. Google invented this technology because indexing the Web was infeasible with existing systems. Now companies adopting Hadoop are bringing cloud architecture into their data centers.
The simultaneous rise of cloud and big data technologies isn’t coincidental. They’re mutually reinforcing. The expansion of the cloud continues to drive both the creation of new big data technologies and big data adoption by making it easier and cheaper to access storage and computing resources. Big data infrastructures also play a role in this trend. For instant, recent advances in the Apache Hadoop ecosystem enable more types of workloads and more tenants to share a cluster. Users will be able to run more types of infrastructures on the same Hadoop cluster, which itself may be running on a cloud infrastructure. As big data infrastructures become more generic, the cloud infrastructure will add more
specialized services for data storage, processing, and analysis.
The combination of big data, cloud computing, and new algorithms and techniques for visualizing information enables converged analytics—performing analytics on data from many different sources. These new techniques for data delivery and data management also enable cloud-based Analytics as a service (AaaS).
From security and privacy to pricing models, the combination of big data and cloud computing is having a substantial impact on the nontechnical aspects of our lives as well. There is a tension between our desire for converged analytics and cloud computing, which is about sharing more computing resources and data with increasingly diverse tenants, and our desire for better privacy controls and data protection. Usage-based pricing models are forcing us to rethink how we produce and consume technology. Future columns will look at how policies and economics are being shaped by these technological advances.
Future columns will examine how people are using these trends together, big data developments from cloud builders, and how people are making the cloud better through data.
– ChauNH2 –