How data migration can be made seamless

How to make data migration can be made seamless

Having worked in Financial Services technology for decades the one conclusion I have come to is that legacy systems cannot be avoided.  Most companies have them.  Even newer companies have systems that are problematic.  Data quality is the bane of most companies lives.  Understandably then, when TAINA talks to potential customers about implementation, migrating data away from legacy systems and addressing data quality issues is often one of the key fears.

TAINA has invested a great deal of time and effort building tools and processes that make transferring data away from your legacy systems and improving your data quality. In this article I will explain how TAINA achieves this.  We have a tried and tested approach which has been proven to work with a number of legacy systems and some very awkward data quality issues.

TAINA’s 7 Step Data Migration Process

TAINA’s Data Migration approach involves seven steps to success, all of which are outlined below. 

1. Initiate the Project

This may sound obvious, but you need to treat the data migration process as a project.  It needs people to be formally allocated to the project with associated roles and responsibilities, it needs to have clear goals, clear measures of success, a clear communication strategy and clear timescales. 

Just to be clear, the TAINA migration tools make the process of migrating data quite straightforward, so we are not saying you need to have lots of resource allocated, but you do need to have clear decision makers for example.  Let’s say some data quality issues require decisions to be made which impact your data then you need to have an internal expert who can make the best decision for your organisation.

2. Analysis Phase 

The first actions involve analysing the data you do have.  For example, how many customers or accounts, how many different types of tax forms do you need to migrate, what are the various tax classifications that you use.  Even more fundamental, which data do you wish to migrate?  TAINA has worked with customers to migrate varying levels of data.  For example, some just wanted to migrate their customers, some their tax form data, others just wanted to migrate the tax form document itself.  TAINA can accommodate all of the above and more.

3. Preparation Phase.

Once we are clear on what data is to be migrated, and what the shape of the source data looks like, then we need to prepare for data migration.  This often involves technical tasks such as launching servers, but TAINA will also undertake all the data mappings for you.  This means TAINA will often make some simple changes or configuration updates to its data migration tools.

4. Dummy Migration Run(s).

Before we launch into a full migration, we have found that the best approach is to migrate a small subset of data.  This data should be a representative sample and should be low volumes.  Once you get into the migration properly, then problems are harder to identify and resolve quickly.  By starting with a small scale set of data you can iterate very quickly.

The best way to be confident that the migration will work is to ensure you can run a full end to end migration, with a small dataset, successfully.

TAINA’s migration tools will highlight all problems with the source data, the input files, the migration app setup and configuration or the data mappings.  The reports are simple to understand, and TAINA will help you identify problems.

5. Run the Migration.

This is the main phase of the project.  TAINA can migrate data in a ‘full’ or a ‘delta’ mode.  By running a ‘full’ migration then you can have confidence that we are starting from a clean slate because the migration tools will clear out all data in the target databases first.  A ‘delta’ migration will allow you to run incremental migrations, building up the final data set step by step.

TAINA will provide comprehensive reports to show you exactly how effective the migration process was.

In our experience this phase will be highly iterative.  It is not uncommon to run this process five or more times, each time improving the effectiveness of the migration process.  Being able to clear out the target system without human intervention makes iterations very easy to run.

After each migration run TAINA will provide detailed metrics.  We will be able to show you what has migrated, what hasn’t and what partially worked.  The key to making the process effective is being able to provide detailed measurements for the extent to which migration worked.

6. Final Validation.

Once the migration process has finished and you are happy with the data reports TAINA has provided, then it is important for you to confirm the migrated data will work for your key user scenarios.  TAINA will have helped you identify what these scenarios are during the initiation phase.  The purpose of this phase is to ratify that the system is now working as expected, from start to finish.  This doesn’t need to be an onerous task, but it is a vital step.

7. Signoff.

When we get to this point all the hard work is done.  TAINA often gets asked to run these migration projects in a short space of time, but the purpose of this phase is to take a step back and review the success criteria which were set out at the start of the project.  The senior stakeholders need to be satisfied that the data is now transferred over to TAINA correctly and that the new TAINA system has been commissioned successfully for operational use.

6 Key Lessons Learned

TAINA has learned some difficult lessons over time, having been through this process many times.  In order to help you avoid the same mistakes we would suggest that you take note of the following.

1. You Need Senior Stakeholder Buy In

Sometimes you need to take difficult decisions, especially if you have significant data quality issues to overcome.  Having your senior stakeholders bought into the data migration process really can help move obstacles in your organisation should you encounter any.  By keeping senior stakeholders close to the project throughout can only be a good idea.  TAINA will help by providing regular updates.

2. Start by Fixing Data at Source  

The old saying ‘rubbish in, rubbish out’ is so true.  The TAINA migration tools will identify any data quality issues you have.  TAINA can even help you correct some of the data dynamically where possible.  However, experience shows us that it is prudent to resolve data quality issues at source where you can.  Being able to easily run the TAINA migration process time and time again really helps with this.

3. Make it Measurable

The only way to run a successful data migration process is to make every step highly measurable and highly objective.  Knowing exactly how many records worked perfectly, how many addresses were incomplete, how many dates of birth are in the wrong format, etc, etc. is the only way to be sure that the migration is configured correctly or that changes to source data are effective.

4. Define Success Criteria

If your data quality at the start is truly, truly awful then it is highly unlikely that you will end up with perfect data at the end.  It is not impossible this will happen of course, but there can be a point of diminishing returns to fixing data issues.  If you can get to 99% accuracy in two weeks, but the estimate is six months before you get to 100% accuracy then your organization needs to decide is this is worth the time investment.  TAINA will be there to help you, but having clear success criteria can really help with the decision-making process during the project.

5. Detailed Validation of the Migrated Data is Key

TAINA will provide a detailed set of validation scripts and reports at the end of each migration run.  We have seen customers accept the fact that all customer records migrated successfully at face value, but what if a few of the records had warnings and one of the low-level data points was missed off?  It is important not to underestimate the importance of really digging into the migrated data and being certain that the data is correct post-migration.

6. You Will Always Have Outliers!

‘Rubbish in, rubbish out’.  This will always be the case.  TAINA has not seen one migration project without some awful data points being found.  It really helps to accept this fact and decide upon the extent to which you are looking for perfection.  TAINA will of course help you at all times but be prepared for the long haul if you are wanting to get to absolute perfection.

TAINA: Data Migration made Easier

We speak to a number of prospects and customers in the Financial Services space, and almost all of them have a fear of migrating data. TAINA has taken that on board and has created a set of processes and tools that make migrating data so much easier than you may think. 

We have learned many lessons over time and gained enormous experience in this area. 

We have dealt with many scenarios, from the small startup through to large multi-national banks.  Whilst the scale of the data may vary the problems are quite common and we believe we can bring enormous value to your organization.

We would love to talk to you more about implementing the TAINA Platform, or for more information on our data migration process  get in touch or request a demo