Data Migration in 6 Steps

Data Migration Definition

Data migration is a project where we move or copy data from System A to System B, and remove or decommission it in System A, like in application migration, storage replacement, systems or applications upgrades, disaster recovery. It should be as automated as possible, freeing up human resources from tedious tasks.

Migrating data is the process that includes preparing, extracting, transforming data from source systems, and storing data to the target system. Validation of migrated data and final decommissioning of legacy data are also part of the entire process.

We also have to understand that data migration is not data integration but is one of data integration styles. and usually represents a one-time project of moving the data between systems. A clear example is when a Company is re-platforming its core ERP system, for example from Navision to SAP. When re-platforming occurs, migration is necessary. Organisation will migrate the data to the new system because the original system will no longer be used or maintained. 

Data Migration Challenges

Going back to our ERP example above, let’s imagine each system as an individual library; each book in each of those libraries is your data. The library has shelves spaced at specific heights, but the new library doesn’t use the same distance between shelves. And in order to get all the books on the new shelf requires re-organizing the books entirely.

This re-organization of books (i.e. data) becomes the primary challenge of migration. Placing the data without any sort of data transformation can result in mixed up, incorrect, or missing data altogether.

So migration has a very specific purpose when you absolutely have to move your data permanently. For example, your existing ERP system is shutting down and you have no choice but to get your data onto another platform. In cases like these, migration is your only option. In other cases, data integration is more common approach.

Data Migrating Strategy

There are many ways to build a data migration strategies and companies will select the most appropriate one according to their specific business needs and requirements. However, most strategies fall into one of two categories:

• Big Bang
  Big bang data migration is an approach, where whole transfer is completed within a limited time window. Live systems are while data goes through data processing and moving to the new database.
  The advantage of this method is, that it all happens in one relatively short time-frame. The pressure on the other side can be significant, as the business operates without one of its resources, which risks the implementation.
  If you decide the big bang approach, be sure to run through the data migration process before the actual event.
• Trickle
  Trickle migrations, in contrast, complete the migration process in phases. During implementation, the old system and the new are run in parallel, which eliminates downtime or operational interruptions. Processes running in real-time can keep data continuously migrating.
  Trickle data migration can be fairly complex in design, but if done right, usually reduces risks.

Six Steps of Data Migration

Strategies vary in the specifics, but generally, a data migration plan has to follow common steps:

1. Explore and Assess the Source

You must know and understand what you’re migrating, how it fits within the target system, how much data is pulling over, and what that data looks like before migrating data. You should identify unnecessary fields in source systems data, missing data fields that should come from another location to fill a gap.
Then, you should profile and audit the actual data from sources. Poorly populated fields result in incomplete data pieces, inaccuracies, or other problems.
If a company skips this source review step and relies on data as is, it can waste time and money on migration process, or even run into a critical flaw in the data mapping.

2. Define and Design

The first step of the design phase is to select the type of migration – big bang or trickle, and based on that define the technical architecture of the solution and detail the migration processes.
Well documented project plan and associated risks can be prepared on the gathered information about source data and the target system. A project plan should also include a security plan for the data that needs to be protected.

3. Build the Solution

An attractive trap for companies is the “just good enough” approach for data migration. However, it’s crucial to get it right, because you will only perform it once (hopefully). A common tactic is to break the data into subsets and build out one category at a time, followed by a test. If an organisation is working on a particularly large migration, it might make sense to build and test in parallel.

4. Test, test, test

The testing process isn’t just over after testing the code during the development. It’s critical to test the migration design with real data to ensure the accuracy of the implementation and completeness of the application.

5. Turn the switch 

After final testing, turn the switch on and proceed with the live data migration according to the defined plan.

6. Audit

After data migration, check the migrated data with an audit system in order to ensure the accuracy of the migration.

One of the largest data migration projects in Slovenia, that includes data from numerous transactional systems into SAP is performed by CRMT with Talend Data Integration platform.