Data in the Cloud
Curious Joe recalled the earlier insightful conversations with Genius Jane when Diving into Data and zooming in on Data Lakes.
Joe was aware of the “Build it, Secure it, Scale it, Maintain it, Migrate it, Move it, Close it” paradigm of data centers housed within owned premises. Everything needed to be taken care of, from building resilient infrastructure, ensuring stable & sufficient power supply, adequate cooling, connectivity & access controls, provisioning storage capacity, maintaining back-ups, ensuring fail-over to back-up data centers, taking care of patches and upgrades, and having people around 24x7 to take care of the complex. This seemed too costly besides coming along with sub-optimal usage of resources, being possibly risky to operate & susceptible to cyber-attacks, and primarily being a non-core business activity.
Curious Joe heard about the Cloud lately and Genius Jane had promised to tell him more about it. “If data centers ultimately capture and store customer data …”, he wondered, “ then how does this stack up in terms of data in the cloud?”
Good for him that he came to know that Genius Jane was traveling in the same private jet that he was scheduled to get on board. Both had received an unexpected call from the very higher-ups to be at another city that afternoon for an urgent meeting. They got very lucky with the travel arrangements. This was also a perfect setting to speak about the Cloud up above the clouds!
Curious Joe: “Hey Jane, could we catch up on the topic of cloud & data in the cloud? Tell me all about that!”
Genius Jane: “Well, Joe, let’s start first with cloud infrastructure. Rather than own all the infrastructure, anyone could literally sign up for infrastructure and pay a monthly rental for it. Cloud providers are in the business of providing infrastructure as a service and you only pay for what you use. Provisioning more capacity is a matter of a few clicks in the console UI and you usually get what you demand in a few minutes. You can not only scale up but also scale down in terms of the infrastructure requirements. Someone else can use that freed up compute or storage capacity. This is handled automatically by the cloud vendor.”
Curious Joe: “A few minutes? That’s fast! I see how this could be optimal as well. So, how about we talk about Data in the cloud?”
Genius Jane: “Curious as ever, Mr. Joe! We have schema-on-read and schema-on-write… structured and unstructured data. They can be uploaded to cloud-based object storage repositories or streamed directly to it; as well as into managed RDBMS, No-SQL Databases & Data Warehouses. In the contract with the cloud provider, the customer owns the data including managing encryption — at rest, in-use and in motion, as well as being in charge of taking care of the backups & replications. The cloud vendor would take care of maintaining the underlying infrastructure. That’s huge outsourcing of traditional in-house work of the privately-owned data centers. Best of all, the cloud vendor has no way to access your data, and reputationally would be detrimental to do so. With streaming data that grows over time or having sporadic spikes in the incoming data, the elasticity of the cloud makes it easy to add more resources quickly as needed. This prevents potential outages from occurring due to resource exhaustion.”
(Photo by Sigmund on Unsplash)
Curious Joe: “So, let me see if I understand this right, if I want to test out a new digital product on the market and provision end-to-end scalable infrastructure all the way from the front end APIs to the back end data stores, I can just do it. If the product is a hit with the customers, I can scale the infrastructure to handle more traffic from customers. If it flops, I can just shut down the provisioned infrastructure in the cloud and prevent further costs from incurring, right?”
Genius Jane: “That, and also you can backup and download the data already collected for archival purposes if you need to.”
Curious Joe: “I see how this helps with agility as well. No need to wait for months for new compute and data capacity to be installed in order to launch a new idea. You had also mentioned leveraging the data for analytics. Can that be done in the cloud?”
Genius Jane: “Sure can be done! Cloud providers offer tools to extract, load & transform the data. These tools are often based on open-source tools. And then data science activities can be performed and machine learning algorithms can run on the transformed data to provide predictive & prescriptive analytics to the business. I mentioned setting up data warehouses and data lakes on the cloud. That is also possible. And there are also data visualization tools available. So you can do pretty much everything with your data on the cloud these days. There are also new forms of data organizational structures being talked about, such as data mesh. A new variant of Conway’s law materializing here possibly.”
Curious Joe: “Brilliant! It’s been a pretty smooth flight so far. The pilot has announced that we are going to the land shortly. Let’s get back to our seats and belt-up. I learned today that we can get data in the cloud and get up and running in a short timeframe without really needing to worry about the underlying infrastructure. The time elapsed from data capture to actionable insights get shorter and shorter by leveraging the cloud. Can we talk about the path from data to insights in more detail the next time we have a chat?”
Genius Jane: “Sure, let’s do that! First, let’s see what the urgent meeting called for is all about. Looking forwards to the return flight back home later tonight.”
Curious Joe: “Me too! Wow, look! there’s the city :)”
(Photo by Pedro Lastra on Unsplash)
