The ERP data migration conundrum
The key challenges and confusions in the ERP data migration journey are:
- Defining migration strategy – Defining strategy for extraction of data from different source systems, cleansing of data, validation and reconciliation of data, environment planning is very tricky and needs a customized approach suiting the project requirement.
- Finalizing tools and technologies for migration – Do I really need to invest on a specific tool for migration? Or, will an Excel-based approach suffice? There are many tools available in the market. Which one will suit my requirement? In the absence of right tools, automation, repeatability and reusability will be missed, thus increasing the budget of the project.
- Identifying source of truth for migration – Do I need to consider CRM system as the “Source of truth” for customer master data or can we migrate it from the legacy ERP system. In case of conflict, what will be the approach to resolve the gaps?
- Finalizing the history of data to be migrated – What is the amount of history we want to migrate to the Target ERP system; Can we run the operations without any history, or should we migrate one year of transactional history or the complete history?
- Master data harmonization and data cleansing approach – For a multi-country rollout scenario, same master data can come from different countries, source systems or we can have duplicity in master data within the same country. What should be the approach to harmonize the master data? How will we identify the data gaps between source and target ERP systems to provide an improved customer experience and remove existing operational inefficiencies?
- Sensitive data handling – What will be the approach to handle sensitive data during the migration process? Are we going to use any specific tool for that? How are we going to ensure adequate data testing as well as remain compliant with regulatory requirements?
- Cut-over approach – How are we going to ensure the accuracy of data being migrated? When and how will functional testing be performed on the migrated data? What will be the cut-over approach to have minimal impact on business as usual? Can we have zero downtime? If not, what is the optimal window we should plan for shutdown and how to manage the business continuity during the outage?
- Data mapping - Data mapping between source and target system is the key for the accuracy of migration and at the same time, a complex and time-consuming activity, especially for the scenarios where there are heterogeneous source instances. Complexity further increases with the level of customizations across source systems and availability of source system knowledge within the team.
- AS-IS vs TO-BE gaps and localization handling – What is the impact on data in case of process and functional gaps between AS-IS and TO-BE? How do we handle data for localization changes?
Resolving data migration challenges
The key to addressing ERP data migration challenges is building the right focus from the beginning by establishing a dedicated migration team involving like Source, Business, Application and Technical SMEs.
The apt approach and best practices are:
- Migration strategy - During the early phase of migration, Migration SMEs should work with the customer IT team to define the migration approach. Approach around data extraction, profiling, transformation, cleansing, loading and reconciliation must involve all stakeholders. During the build phase, data can be extracted from non-production environment, but production data may be needed during Mock runs, to identify and address the actual data gaps. In many cases, a copy of production data is provided to Migration SMEs in some landing area to avoid impact on BAU. As a best practice, data in the staging area should be pulled without any filtering. This helps in reconciling the data better as production data changes. For loading of data in Target, target-specific APIs / utilities should be used. Business SMEs should perform business reconciliation on migrated data.
- Tool finalization for migration – Tool must be planned for the migration. This ensures repeatability, consistency and automation in the migration process, which is a must for large scale multi-country rollout migration. Investment on the tool provides faster returns but selection of tool is always confusing. Depending upon need around data quality, size of the market, data volumetric, number of markets, source and target systems, different tooling options can be explored. Every tool has its own advantages and disadvantages. The Architect team of the organization should evaluate different parameters and the organization’s requirement to find a suitable balance.
- Application mapping – In the early phase, the Data Migration team always faces a common challenge – Where to start, how to start? The starting point for the Data Migration team should always be the TO-BE Functionality. From there, they can start mapping the source applications catering to these functions for the specific market or plant in scope. If there are multiple countries, markets, plants in scope for a specific rollout, this exercise needs to be carried out for all the units. Next step would be to identify the data objects for target specific modules and start mapping the source of truth for all data objects. After completing this exercise, the Migration team should create a data mapping plan by answering the key questions as depicted in Figure 1.
- Data history – To enable a business in taking the final call around history, we should ask questions like - do we have history data available in any other system? Is this needed for operational or regulatory purpose? Details relating to frequency and users referring to historical data? and can we plan to archive or push the historical data in some other data repository? Active Master with open balances is the best approach to avoid post migration reconciliation issues. Majority of ERP migrations follow this approach to ensure hassle free reconciliation of data. Data warehouse or other reporting systems can be referred for historical transactions.
- Data harmonization and cleansing – Data Migration team should plan early profiling of source data to identify data gaps and gaps should be discussed with business / data owners to finalize the remediation plan. Business should plan / factor efforts for cleansing the data in source systems and at the same time, some standard rules can be provided to standardize the data during the conversion. As part of the data quality approach, another important area is to de-dupe master data. This is very critical especially for multi-country consolidation. As a standard practice; master data should be de-duped and harmonized both within the source data and from the target ERP instances. If there is duplicate record within the same source system, the master data is merged during migration process with the help of business SMEs and Master-id in transactions are updated accordingly. In case the master data is already available in the target ERP, it is extended for the current Business unit / Geography instead of new record creation for the master data.
- Policy and strategy to handle Sensitive and PII data like customer, vendor and employee’s personal information or products pricing details etc must be defined as part of migration strategy and strict adherence to the policy should be ensured to avoid non-compliance as it may lead to serious damage in terms of penalty and reputation loss. Tool or script based anonymization / masking of data will serve the requirement.
- Cut-over is the most critical activity of any transformation program and requires proper communication, planning and co-ordination amongst different stakeholders. It is also essential to execute cut-over activities as per agreed plan to ensure a seamless go-live. Typically, master data is migrated in advance and weekend is used to migrate the open balances during the cut-over (outage) window. Smoke Testing and financial reconciliation is performed before starting operations from target stack to avoid post migration surprises.
- Data mapping many a time is considered a simple table and column level mapping between source and target and assumed as an IT activity, while in reality, it’s a business-driven activity involving multiple stakeholders. Who, When, How and What for the Data Mapping activity is shown in Figure 1.
