You often hear others say “If I could do it all over again ...”, or “If I’d known sooner... “.
And that’s certainly true for data migrations. For many organisations, data migration is a one-off event. And it’s exactly those first instances that you are caught in those traps and subsequently - but understandably - make ‘rookie mistakes’.
What are the main pitfalls of data migration?
- “Data quality falls outside the scope”
By far the most significant pitfall is ignoring data quality. If issues do not manifest themselves until the acceptance tests in the new system or, worse still, when you’ve started the production process, it will cost an unnecessarily high amount of time and money to resolve it (if it can be resolved in the first place). Data quality should be kept within scope! And if you’re dealing with all this data now, isn’t this the perfect time to do something with it?!
- “We do have to speed up the conversion software later on”
This is another pitfall. You often hear of the conversion of a small set of data that takes hours. The poor performance of conversion software is not only a risk for the final conversion. How many days will the business be out on its ears? It also bugs you when you’re developing conversion software. You want to be able to test the conversion software several times a day. Often, the underlying cause is that the software is record-based (on the basis of triggers, for instance) and that this cannot be tuned sufficiently later on.
- “We make a golden copy of the source data”
The basic approach to migration should be to extract the data from the source system, to convert this data on the basis of the conversion software and then read it into the new system. And that should be a repeatable process that can be checked. Unfortunately, people still use the ‘strategy’ to make a golden copy and keep that up to date throughout the migration process so that ultimately, it forms the basis for the new system. It's time-consuming and virtually impossible to keep it in sync. So don’t do it.
- “Just look at the source data we received”
For developers of the conversion line, it is vital to know how exactly the source data is presented. Will we be getting mutations or everything? Is it a fixed format or is it separated by special characters? Do the files contain headers? How often do we get fresh data? Can you guarantee any data submitted later is of exactly the same format? If no clear answer can be given to these questions, you’re guaranteed to be facing a challenge, to say the least. Unclear agreements about content and format of data supplies always come with a lot of headaches. So, make clear agreements. Better still, unambiguously specify what, how, when, where and by whom data must be supplied.
- “We’re not yet sure about who will approve the conversion”
One of the first steps of the migration process is to determine the acceptors and acceptance criteria. Who will be approving the conversion? And on the basis of what? And how can you build up and validate your argument throughout the migration process? Developing and testing the checks is as important as the conversion software itself. In the final stages, it often becomes clear that a migration obviously cannot be accepted but the production process is started anyway. What else can you do?
If you’re about to start a data migration, all the best but beware, you won’t be the first ...