Before & After Data Ecosystem Transformation
Data Ecosystem Transformation – Ecosystems are an interacting community and their physical environment. The parts of the data ecosystem include data source, data cleansing, data storage, data science tools, as well as analytics tools.
Years ago before cloud computing rise many business organizations had their own data from on-premise systems. Many executives believe that by maintaining a close IT environment makes their data safer.
The way company executives make reports is by contacting their IT staff or engineers to dump the data into a specific folder. The executives then convert it into an Excel file, cleaning unnecessary data to generate reports. Just exactly as depicted in the chart below.
The Need for Transformation
As technology evolved from on-premise to cloud computing where it offers high scalability while low cost, cloud computing gains acceptance of many business organizations. Data is also flowing from one business organization to another by utilizing Application Programming Interface (API) where it makes it possible for outsiders to transact with the company. The data is now ubiquitous.
Coupled with business needs that demand high speed by requesting real time data and need for analytics tools and data science tools make the ecosystem of data transformed, as follows.
As seen in the second chart, data sources are varied from cloud data, local databases, such as ERP data or Core Banking System Data, and also Rest API. The entire data can be generated real time and stored in a raw storage such as Hadoop.
Hadoop is a framework that allows for the distributed processing of large data sets across clusters of computers. It has two primary components: HDFS, a distributed file system, and MapReduce, a system for parallel processing of large datasets in HDFS.
As usual, the data needs to be cleaned and subsequently stored in the data warehouse for being the input for Data Science & Analytical Tools.
Author: Hendrix Yapputro M.Sc. Certified IT Architect, General Manager PT Equine Global