Key Components of a Best in Class Cloud Based Analytics Solution Architecure
Reading time: 4 mins
by Kumar Singh, Research Director, Automation & Analytics, SAPinsider
The future of Analytics is in the cloud
If you follow the analytics space, you are cognizant of the fact that more and more vendors are moving their products to the cloud, building cloud native products. Due to the flexibility, scalability, agility and the ease of building ecosystems that a cloud platform provides, in my opinion, it is inevitable that the future of analytics platform is in the cloud. The exponential rise in edge devices and the need for near real time data streaming also highlights the challenges that on-premise architectures will eventually run into. All the signs point to the cloud !
However, as businesses think about transitioning to the cloud, a key aspect that they need to think about is- what is the best in class aproach of building analytics solution in the cloud. There are various cloud platforms available these days and they provide the key specifications that you would expect from a leading cloud platform (examples are Amazon Web Services (AWS), Microsoft Azure, Google cloud). The key therefore is to understand how you can design the solution architecture in a way that it will deliver the value that you were expecting from transitioning to a cloud platform.
However, as organizations move from on-premises architecture to hybrid and pure cloud based architectures, they also need to understand what are some of the key components in their cloud based analytics solution architecture that they need to think about. In the subsequent section, we will discuss six key components of a best in class cloud solutions architecture. All leading cloud solutions providers share their own best practices documents and you will find these six areas as consistent theme in their documents. For this article, I will leverage AWS Analytics lens, a best in class architecture framework developed by AWS, and simplify it to walk through the architecture components.
The Six key components of a cloud based analytics solution
To simplify the components, let us follow the high level analytics process in the cloud. That process flow is not very different from any on-premise analytics solution process flow. Following are the six key components in the Analytics solution flow in the cloud:
- Data collection
- Data storage
- Data discovery
- Data processing and analytics
- User access
- Data security and governance
Now let us review these components at a high level:
Data collection component: This deals with collecting data from multiple sources, with varying periodicity (like batches or near real time). Examples of sources may range from website clicks, transactions and socia media streams to on-premise and cloud native data sources. The data is typically stored in a central data storage like a data lake or the data may feed into a data hub (my preferred recommendation). An AWS environment example of data collection component is Amazon Kinesis.
Data storage: Data collected is generally stored in a central data storage which then holds and manages the data in order to make it available for downstream applications. This generally is a part of the data lake and supports data files of different formats and different structures. Elasticity and security are two additional key aspects of this layer. Amazon (S3) and Amazon Elastic Block Store are two examples of central storage in AWS ecosystems.
Data discovery : Imagine that you have built a central storage in the form of a data lake. Considering that you have data files being dumped there from multiple sources in multiple formats, it is imperative that you have an automated way of data discovery in your cloud architecture. AWS Analytics lens defines this as “the catalog and search layer” that pertains to discovering and cataloging the metadata pertaining to your data assets.
Data processing and analytics: From the perspective of analytics, this is where the real action happens. This layer enables the querying and processing of data that allows us to generate insights from the data stored and discovered. Each cloud service provider offers a gamut of products that can be leveraged in this layer. These products are often aligned to the type and format of data stored in the storage layer. Exaples for AWS environment are Amazon Athena, Amazon Redshift and Amazon Sagemaker.
User access: This is the front end of the architecture which the user will leverage to access the analytics solution. The critical aspects of this layer are ease of access, secure access and access management. Again, depending on which platform you are on, there are several options available to configure this layer. Some examples from AWS architecture are AWS Lambda and Amazon Cognito.
Data security and governance: One of the most important components/layers, this is the key towards successful adoption of cloud since data security is cited as the top apprehension executives have when citing bottlenecks towards transitioning to public cloud. This layer or component is responsible for ensuring that the data can only be accessed by users who are authorized to access the data. This layer essentially creates a secure environment for the entire architecture. AWS Identity and Access management and AWS Key Management services are widely used in AWS ecosystems.
What does this mean for SAPinsiders ?
- Cloud based analytics solutions provide many advantages over on-premise solutions however, designing an optimal architecure is critical. While the article explores the components of the architecture at a high level, you have to ensure that the architecture encompasses the current requirements as well as is designed to accomodate any future roadmaps.
- Azure, AWS and Google cloud, all provide robust and best in class components to build your architecture but you have to keep in mind that the technology is just an enabler. It is the “art” of business strategy, technology strategy and technology architecture that will help you build analytics solutions that align with your business imperatives.
- Your entire end-to-end architecture need not be on the same cloud platform. An example, you may have cloud based data sources and application on one platform, that then connect to SAP cloud platform, where you can leverage SAP analytics cloud. If you have the vision and creativity, technology options galore !
Kumar Singh, Research Director, Automation & Analytics, Supply Chain Management, SAPinsider, can be reached at email@example.com