Press "Enter" to skip to content

Data Migration – Does it Need to be Scary? by Mark Lloyd, Preact

Your business’s data is one of its most precious assets, but when it comes to data migration a lack of planning and the ‘fear factor’ can become pain points for organisations.

After all, it’s your intellectual property which needs looking after and nurturing so when you come to move it, why not treat it as a valuable commodity rather than an afterthought?

This is according to  , who has penned the thought leadership piece below on data migration and how planning ahead is the key to a successful transfer.

When it comes to big data, especially when it’s structured, many organisations lack the experience and knowledge to carry out a successful migration.

“A lot of people have never lifted the covers or even thought about how their data is stored and structured – and why would they unless they’ve actually built one of these databases,” said Mark.

“Unless you’ve got a benchmark to go against, sometimes there’s a fear factor about data – especially data migrations. Even if you’ve never done a migration yourself, you probably know someone who has, and it’s often got this kind of scary vibe to it.

“But you probably don’t know why it’s scary because no-one’s ever sat down and told you.  Probably because the people with those data migration scars and bad experiences, spreading that fear, don’t understand what went wrong – how they could almost certainly have had a much smoother migration if they had planned appropriately.”

Just as many people are fearful of data migrations, Mark believes others consider this will be quick and simple, “because we all move data every day – whether it be a WhatsApp file or an email attachment.  Moving data between databases is more complex, though and requires appropriate preparation.  Underestimating that is probably the root cause of the migrations that have ended up being painful – which then drive the fear factor as scare stories.”

When it comes to these key pain points of data migration, Mark uses the analogy of moving house to explain the issues many people face.

“You need to think about what you need to do long in advance. Do you leave packing to the day before the removal van is coming or do you try to spare your sanity and plan ahead?

“The answer is to start as early as possible. Do little things often and use the opportunity to get rid of items that aren’t needed. Be brutal, especially given data retention legislation and responsibilities.

“For each hour spent on planning, many more will be saved during delivery.

“Data migrations are a great opportunity to put procedures in place and establish best practise that will benefit you moving forward, making your company more efficient and hopefully more compliant as well.”

Mark Lloyd, Head of Data Services at Preact writes:

In the IT world there aren’t many phrases that can put a damper on a conversation as abruptly or as effectively as “Data Migration”.

Those two apparently innocuous words all too often cause even the most hardened and seasoned of IT professionals to break out in a sweat and rapidly circle around the subject to something, anything else.

Or, perhaps even more unhelpfully for those that have never been involved in a Data Migration before, a colleague may pull a grimaced expression on hearing those words, then swiftly exit stage left with a pat on the back and mysterious, ominous “good luck!” – leaving them even more in the dark and with an all-new concern that something unpleasant may lie ahead. But why is this the case and is this fear of Data Migration justified?

Data Migration is often scary because it is a task that often catches people out. Many can see it simply as a case of moving data from A to B, which essentially it is. However, the amount of preparation that is required to achieve a smooth migration is often underestimated and data is not forgiving. There isn’t much room for error – the migrated data is either correct or it isn’t. The ever-increasing volumes of data that needs to be migrated isn’t making the task any easier and the regulatory responsibilities placed on data administrators, by the likes of GDPR, only add to the pressure. The migrated data must be correct.

An added challenge for those managing Data Migration projects is that there are often many unknowns. Your data may appear one way in the user interface but behind the scenes, in the underlying database from which the migration will read the data, the data can be stored, and even named, completely differently.

In the UI an “Account” form may suggest the data is all from one unified table, but the reality may be that it is in fact a composite, from many different tables. And a field with a friendly UI display name, such as “Territory” may have a database schema name that is totally different and obscure, such as “UD_Text1”. So, one of the first tasks is to identify which fields from the UI need to be migrated and to then ascertain and document where these are located in the database. Often organisations are not in a position or experienced enough to do this, and the source system may be bespoke, old or both – which creates an added challenge for an IT Partner to navigate and reverse engineer – certainly alone.

The analysis must be a team effort, with the Client, as the Data Owner, responsible for clarifying what the data is, what needs to be migrated and how it links together in the UI – with the IT Partner helping them, if required, to translate that to the database.

The “Four Pillars” of Data Migration

Due to the many variations in source and target system – and the types and quantities of data to be migrated – each Data Migration can feel like a first, which in turn can contribute to the fear and wariness mentioned above.

But rather than focusing on the differences, it is more constructive to focus on the constants – what factors or steps will always be the same? Taking this approach can lead to quick progress and can help break scaling the mountain down into manageable tasks.


  1. Access The Data

A seemingly obvious requirement, which often causes delay, is that analysing and migrating data requires access to it. Whether the data is in a spreadsheet, an on-premise database or cloud-based system, IT Partners will need read-only access to this – which can take time to provision the appropriate credentials, VPN access etc.


  1. What Data to Take

As the Data Owner, only the Client can determine what data is needed for their business, what data can legitimately and legally be retained, and what data, if any, can safely be left behind in the legacy system. The earlier internal conversations can be had to try and prepare answers for these questions, the better.

Also, striving for a “minimum viable” dataset nearly always yields the most successful migrations. It is too easy to say “take it all” – often through fear of leaving anything useful behind – but a Data Migration is the prime opportunity to “clean house” and populate the target system with only the best and most useful data that follows the principles of the business’ Data Retention Policies. What’s that – you haven’t formalised any, yet? Again, this is the perfect opportunity to do so – not only to ensure the target system is populated appropriately but also so that measures can be put in place to keep the data compliant.


  1. Where To Find It

Having determined what data is required the next task is to identify where to find it. This could be columns in a spreadsheet or fields on the source system. For the latter, initially identify and document where the required data can be located in the front-end – then assistance may be required to map out where the data is behind the scenes, in the underlying database.


  1. Where To Put It

The final piece of the puzzle is to identify, in the target system, where all the data needs to go. Most data might be migrated 1:1 but some entities may need to be split or merged, some option-set values changed or re-mapped and possibly some data may need transforming or cleaned en-route – for example, capitalising the first letter of name fields or removing any text from phone numbers.


With these four elements completed, it should be possible to build a “Mapping Schema” that identifies exactly what data needs to be migrated – where from, where it needs to be put and if and how it needs to be transformed. This is the recipe book for the Data Migration and the key to a successful process that does not need to be feared!

Thinking about a Data Migration?

If you’re about to embark on a Data Migration, or are considering an integration with Dynamics 365, Dataverse and the Microsoft Power Platform, contact us to find out how Preact can help.

Please follow and like us:

Be First to Comment

Leave a Reply

Technology Reseller Magazine & Site is Published by Kingswood Media 2022