100% success is required when migrating large amounts of data. Success is defined by the fact that the data has been transferred correctly and that the target system works as expected.
Still, how can you be sure that the data migration was successful? How do you know that from the set of hundreds of thousands of records that have been migrated not a single one is missing? Or that data has not been exchanged? And how do you guarantee referential integrity? What securities do you have in terms of the financial link-up? In short: how do you prove a successful migration?
Are you involved in a data migration project as a controller or a test coordinator? If so, chances are that you asked yourself these very questions. The 7 proofs set out below can be used in any data migration project in which 100% success is vital!
Learn from our experience!
Below are the 7 proofs in the form of checks and reports that together provide proof of a 100% data migration. These proofs are based on more than 35 migrations.
Only clean data has been migrated
This check concerns the data quality of the source data. This ensures that the target system is filled with correct and relevant information only.
Check per object and per attribute that the quality is sufficient for conversion to the target system. If the quality of a single attribute of an object is insufficient, the transformation of the entire object is suspended.
The data quality report contains all objects the quality of which is insufficient. Use this report to clean the source data.
With every transformation, this check validates whether all attributes have been correctly converted or manipulated.
The format of many data elements is changed during migration, but they do not change intrinsically, such as a customer's date of birth. Sometimes, the contract lines of a customer are merged, but the total contract value before and after conversion remains the same! Check this by comparing the post-migration value of each attribute against the value from before migration.
Ask an expert other than the one who realised the transformation software to draw up these control reports. Use these reports to test the transformation software.
Incorrect keys are often a reason why transformed data cannot be loaded into the intended target system. Therefore, explicitly check the transformations and the generation of primary and other unique keys.
A simple check whether the (primary) keys are truly unique after transformation.
Use these reports to test the transformation software. The cause may also lie in the source data — use these reports to clean it.
In the target landscape, the relationships between objects must be correct and complete. Contracts cannot exist without a customer!
Check for each object whether it has a correct reference to other objects it has a dependency with.
The reason for the lack of referential integrity may lie in the fact that objects have failed during the data quality checks. Incorrect generation of (primary) keys may be another cause. Therefore, use these reports to clean source data or refine transformation rules.
After the transformation, each attribute must meet the data constraints of the target system; perhaps some date fields in the new system may only be in the past, whereas the source did not have these constraints?
Validate these constraints by means of separate checks on the transformed data.
If the cause does not originate from the source data, the transformation rules will have to be refined.
Merely checking that the data has been migrated without loss does not suffice. The fact that it is complete has to be demonstrated as well! Long and numerous discussions with accountants, auditors, process owners, etc. have taught us that two types of audit counts offer a solution: object counts and acceptance counting lists. This evidence is about object counts. The evidence that follows on from that discusses acceptance counters.
Count the number of objects in scope in the source, in the target and at each intermediate step in the migration process.
Object counts provide insight into how many records there are in scope, how many records are validated, how many records have been transformed and the number of records that pass post-transformation checks. The counts must be in succession to ensure that not a single record is lost during the process.
Acceptance counting lists
In addition to the object counts, acceptance counting lists produce evidence of, for example, the financial link-up. The total contract value must not change before and after each step in the migration process. The same goes for accounts payable, accounts receivable, etc.
These counting lists are drawn up together with the customer, specifically from a business perspective such as operational and financial management.
These are often reports that already exist in the source and target systems. By including these in the development of the migration software, you can use these counts for a financial reconciliation between the source and target systems.
If you can, combine the checks and reports referred to above in an automated process that checks source and transformed data on a daily basis. This creates momentum in the project, the quality of data and transformations increases and checks steadily reduce loss. This in turn increases confidence in a successful migration.
The last thing to be demonstrated is that the target landscape continues to operate correctly and predictably after data migration, for example by means of a conversion acceptance test, performance test, chain test, shadow running, etc.
Are you looking for detailed substantive advice for your specific data migration? Please contact us without obligation. We have performed over 35 successful data migrations ranging from mortgage portfolios, investments, corporate and consumer credit, pensions, insurance and benefits.