Optimizing Data Management and Reducing Costs through Migration from Oracle to Azure File System and Azure Blob using Newt Global’s DMAP

Executive Summary:

This white paper presents a successful migration of 70+ TB of time, event, and meta-data from Oracle to Azure File System (AFS) and Azure Blob (BLOB) using Newt Global’s DMAP product. The customer was facing high costs of nearly $100,000 per month to manage the data in Oracle and needed to migrate recent 2 years data (Approx. 20TB) from Azure Hyperscale PostgreSQL to AFS/BLOB. Additionally, the customer wanted to test the API functionality using migrated data on AFS in ORC format.

Challenge:

The customer was facing significant challenges in managing their data in Oracle, including high costs and the need to migrate recent 2 years data (Approx. 20TB) from Azure Hyperscale PostgreSQL to AFS/BLOB. Additionally, the customer wanted to test the API functionality using migrated data on AFS in ORC format.

Solution:

Newt Global’s DMAP was used to assess the current architectural components, including MapR streams and Oracle database configurations, and recommend the migration approach and target architecture on Azure Filesystem with defined directory structure. Leveraging DMAP’s multithread framework, multiple VMs were deployed to retrieve, transform data to ORC file format, and migrate to AFS/Blob. The data migration included approximately 10:1 compression on transformation to ORC file format, generating more than one million ORC files as part of the migration. At scale data validation against source database with row count and averages for select columns was performed, with automated rerun of data load and validation in case of failed validations. Power BI dashboard was used for real-time tracking of data migration and validation, providing greater visibility to management on the progress of the migration. Tested APIs to read from AFS based on certain input parameters were also performed. Backup of all data was taken on Azure Blob.

Business Impact:

The successful migration of 70+ TB of data from Oracle and Azure Hyperscale PostgreSQL (COSMOS) to Azure File System and Azure Blob resulted in significant cost savings for the customer. The migration and modernization were completed on time with zero errors, and the estimated savings from Oracle Migration to Azure by retiring Oracle & MapR was I Million + per year. The customer was able to reduce their costs by nearly $100,000 per month and leverage Azure’s scalable and secure infrastructure for their data management needs.

Conclusion:

This white paper has demonstrated the successful migration of 70+ TB of data from Oracle and Azure Hyperscale PostgreSQL (COSMOS) to Azure File System and Azure Blob using Newt Global’s DMAP. The migration resulted in significant cost savings for the customer, enabling them to reduce their costs by nearly $100,000 per month and leverage Azure’s scalable and secure infrastructure for their data management needs. The solution provided by Newt Global’s DMAP enabled the customer to migrate their data at scale, perform at scale data validation against source database, and test API functionality using migrated data on AFS in ORC format. Overall, the migration to Azure using DMAP provided the customer with a modern, scalable, and cost-effective solution for their data management needs.