Migration strategy for historical data

When performing a data migration, we often get the following question: “Not all data will be transferred to the new system. What do we do with the data that remains?"

A common issue

This is an issue that often arises in data migration. Legacy systems are being replaced and switched off. The current data will be migrated to a new system and the historical data must remain accessible.

It is often a valid choice not to transfer all data to the new system. For example, if the data is not necessary for the execution of automated processes. In addition, you want a lean and mean migration and do not want to unnecessarily burden the new system. The new system may not even be able to handle certain old situations. And, finally, the older the data, the worse the quality often is!

Nevertheless, this non-migrated data often still has to be consulted. Even if it is only temporary. This may also be required to support processes after the migration. Consider, for example, historical payment behaviour of customers. In addition, legislation may require that data must remain accessible for a certain period due to the ‘retention obligation’. For example, a lot of data must be kept for at least 7 years and in specific cases even 10 years.

Document archive as a solution

A historical document archive can be an interesting solution for this. This is an archive in which all relevant data from the legacy system has been transferred to accessible archival documents. This archive can therefore contain all data that was not transferred to the new system during the migration, but that must be (temporarily) preserved.

The document archive can be implemented by a file share, a DMS such as Microsoft SharePoint or a specialised document archiving solution.

Documents are easy to find by searching for keywords, such as names and numbers. Within documents, hyperlinks are used to navigate from the table of contents to the correct text block and to open other documents (for example, multiple contract documents from the same customer) or to navigate to underlying documents (for example, attachments that form part of a contract document).

It is practical to generate a separate document for each main entity within the archive. For example, one archival document per customer or per policy.

Advantages of a document archive

A document archive has many advantages. By adopting the look and feel of the legacy system, the archive is easy to use and requires no additional training for the users. The archive can be accessed via a standard browser. By adding metadata, documents can be found quickly. Often, it is also possible to search for the content of documents using a full-text search engine.

In addition, the archive sets few requirements with regard to management. Authorisation can be arranged by setting different user roles with rights to the folder structure used.

Most document archive implementations support retention policies. A retention period can be linked to each document in advance, whereby documents can be selectively removed from the archive when the retention period of these documents has expired. In this way, the GDPR is complied with correctly - the right to be forgotten.

Note: not always applicable

The document archive is a very interesting solution if the legacy system has to be switched off and all data can be converted relatively easily into a ‘flat’ document structure.

The archive is less suitable (or perhaps not at all) if the legacy screens have a lot of logic. Examples include showing calculated fields in the application screens. In that case, you would have to replicate the calculations beforehand and include the calculated values in the archival documents.